pussycat0x
893d17d413
Multi Port Support Added - JS
2025-12-13 23:11:40 +07:00
Dwi Siswanto
24d1f58682
v3.6.0 ( #6657 )
...
* Multi Port Support Added - JS
* minor -changes
* restoring basic sequential multiport support
* better error handling
* feat(openapi/swagger): direct fuzzing using target url
* fix (openapi/swagger): improve error handling and tmpDir cleanup
* fix(openapi/swagger): err shadowing on write failure
* fix(openapi/swagger): remove discarded error in defer
* fix(openapi/swagger): linter and url validation
* fix(openapi/swagger): remove code duplication
* reusing dialer
* removing debug log
* fix: restore parallel processing in workflow & file proto
add missing `go` keyword to anonymous funcs that
were intended to run as goroutines but were
executing synchronously instead.
Fixes #6492
Signed-off-by: Dwi Siswanto <git@dw1.io >
* test: adds `Test(FileProtocol|Workflows)ConcurrentExecution` tests
Signed-off-by: Dwi Siswanto <git@dw1.io >
* chore(file): satisfy lints
Signed-off-by: Dwi Siswanto <git@dw1.io >
* refactor(integration-test): enhance debug mode detects
* replace hardcoded `DEBUG` env var check with
extensible helper func.
* add support for GitHub Actions Runner env var.
* accept multiple truthy value variants.
Signed-off-by: Dwi Siswanto <git@dw1.io >
* fix(core): race cond in workflow execution
caused by shared context callbacks.
it was exposed after adding concurrent exec to
workflow processing and occurred when multiple
goroutines attempted to write to the same
`ctx.OnResult` callback field simultaneously,
causing data races during workflow template exec.
Signed-off-by: Dwi Siswanto <git@dw1.io >
* introducing workflow sequential mode
* Revert "introducing workflow sequential mode"
This reverts commit 1093bbc62dgit@dw1.io >
* test(core): rm unused tests
Signed-off-by: Dwi Siswanto <git@dw1.io >
* fix(sdk): configure tmpDir for SDK
Closes #6595 .
* docs(sdk): update comment to more accurately reflect purpose
* feat(sdk): add tmpDir configuration option for SDK users
* fix(sdk): init default engine tmpDir when unconfigured
* style(sdk): remove unnecessary else block
* feat(sdk): create parent & tmp dir in WithTemporaryDirectory
* test(cmd): enable `BenchmarkRunEnumeration/Default` bench
Signed-off-by: Dwi Siswanto <git@dw1.io >
* test(cmd): collect CPU & heap profiles
Signed-off-by: Dwi Siswanto <git@dw1.io >
* chore(cmd): satisfy lints
Signed-off-by: Dwi Siswanto <git@dw1.io >
* Merge pull request #6610 from projectdiscovery/feat-result-upload
allow custom id for upload
* feat: write resume file specified by flag
* updating docs
* chore(deps): bump the modules group with 6 updates
Bumps the modules group with 6 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger ) | `1.1.59` | `1.1.60` |
| [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx ) | `1.7.2-0.20250911192144-fc425deb041a` | `1.7.2` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy ) | `0.1.27` | `0.1.28` |
| [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils ) | `0.6.1-0.20251030144701-ce5c4b44e1e6` | `0.6.1` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo ) | `0.2.54` | `0.2.55` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck ) | `1.2.9` | `1.2.10` |
Updates `github.com/projectdiscovery/gologger` from 1.1.59 to 1.1.60
- [Release notes](https://github.com/projectdiscovery/gologger/releases )
- [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.59...v1.1.60 )
Updates `github.com/projectdiscovery/httpx` from 1.7.2-0.20250911192144-fc425deb041a to 1.7.2
- [Release notes](https://github.com/projectdiscovery/httpx/releases )
- [Changelog](https://github.com/projectdiscovery/httpx/blob/dev/.goreleaser.yml )
- [Commits](https://github.com/projectdiscovery/httpx/commits/v1.7.2 )
Updates `github.com/projectdiscovery/networkpolicy` from 0.1.27 to 0.1.28
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases )
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.27...v0.1.28 )
Updates `github.com/projectdiscovery/utils` from 0.6.1-0.20251030144701-ce5c4b44e1e6 to 0.6.1
- [Release notes](https://github.com/projectdiscovery/utils/releases )
- [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md )
- [Commits](https://github.com/projectdiscovery/utils/commits/v0.6.1 )
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.54 to 0.2.55
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases )
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.54...v0.2.55 )
Updates `github.com/projectdiscovery/cdncheck` from 1.2.9 to 1.2.10
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases )
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml )
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.9...v1.2.10 )
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/gologger
dependency-version: 1.1.60
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/httpx
dependency-version: 1.7.2
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
dependency-version: 0.1.28
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/utils
dependency-version: 0.6.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.55
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.2.10
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com >
* refactor(sdk): don't create parentDir when configuring tmpDir
* adding test case
* lint
* removing unused check
* adding multiport template
* refactor test
* chore(deps): bump golang.org/x/crypto
Bumps the go_modules group with 1 update in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto ).
Updates `golang.org/x/crypto` from 0.43.0 to 0.45.0
- [Commits](https://github.com/golang/crypto/compare/v0.43.0...v0.45.0 )
---
updated-dependencies:
- dependency-name: golang.org/x/crypto
dependency-version: 0.45.0
dependency-type: indirect
dependency-group: go_modules
...
Signed-off-by: dependabot[bot] <support@github.com >
* feat(variables): check for undefined params for lazy eval (#6618 )
* feat(variables): check for undefined params for lazy eval
Signed-off-by: Dwi Siswanto <git@dw1.io >
* test(variables): add TestCheckForLazyEval
Signed-off-by: Dwi Siswanto <git@dw1.io >
* fix(variables): fail safe on err compile expr
Signed-off-by: Dwi Siswanto <git@dw1.io >
---------
Signed-off-by: Dwi Siswanto <git@dw1.io >
* chore(deps): bump github.com/projectdiscovery/fastdialer@v0.4.16
Signed-off-by: Dwi Siswanto <git@dw1.io >
* fix(interactsh): skip DNS lookups on interactsh domains (#6614 )
* fix(interactsh): skip DNS lookups on interactsh domains
to prevent false positives.
Prevents nuclei from resolving interactsh domains
injected in Host headers, which would cause
self-interactions to be incorrectly reported as
matches.
Changes:
* Add `GetHostname()` method to `interactsh.Client`
to expose active server domain.
* Skip CNAME DNS lookups in
`(*http.Request).addCNameIfAvailable` when
hostname matches the
`(*interactsh.Client).GetHostname`.
Fixes #6613
Signed-off-by: Dwi Siswanto <git@dw1.io >
* fix(http): prevent false `interactshDomain` matches
Signed-off-by: Dwi Siswanto <git@dw1.io >
---------
Signed-off-by: Dwi Siswanto <git@dw1.io >
* feat: bump dsl with deserialization helpers
* chore: omit unnecessary reassignment (#6622 )
Signed-off-by: ledigang <shuangcui@msn.com >
* disable stale workflow for enhancements
* ci: cache go-rod browser (#6640 )
Signed-off-by: Dwi Siswanto <git@dw1.io >
* chore(deps): bump actions/checkout from 5 to 6 in the workflows group
Bumps the workflows group with 1 update: [actions/checkout](https://github.com/actions/checkout ).
Updates `actions/checkout` from 5 to 6
- [Release notes](https://github.com/actions/checkout/releases )
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md )
- [Commits](https://github.com/actions/checkout/compare/v5...v6 )
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-version: '6'
dependency-type: direct:production
update-type: version-update:semver-major
dependency-group: workflows
...
Signed-off-by: dependabot[bot] <support@github.com >
* do not exempt abandoned issues and prs
* ci: apply free-disk-space on tests
Signed-off-by: Dwi Siswanto <git@dw1.io >
* chore: bump PD modules & update `httputil` calls (#6629 )
* chore(deps): bump the modules group across 1 directory with 11 updates
Bumps the modules group with 11 updates in the / directory:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer ) | `0.4.16` | `0.4.17` |
| [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap ) | `0.0.95` | `0.0.96` |
| [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns ) | `1.0.108` | `1.0.109` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go ) | `1.0.131` | `1.0.132` |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger ) | `1.1.60` | `1.1.61` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy ) | `0.1.28` | `0.1.29` |
| [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx ) | `1.2.1` | `1.2.2` |
| [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent ) | `0.0.102` | `0.0.103` |
| [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils ) | `0.6.1` | `0.7.1` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo ) | `0.2.55` | `0.2.56` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck ) | `1.2.10` | `1.2.11` |
Updates `github.com/projectdiscovery/fastdialer` from 0.4.16 to 0.4.17
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases )
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.16...v0.4.17 )
Updates `github.com/projectdiscovery/hmap` from 0.0.95 to 0.0.96
- [Release notes](https://github.com/projectdiscovery/hmap/releases )
- [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.95...v0.0.96 )
Updates `github.com/projectdiscovery/retryabledns` from 1.0.108 to 1.0.109
- [Release notes](https://github.com/projectdiscovery/retryabledns/releases )
- [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.108...v1.0.109 )
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.131 to 1.0.132
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases )
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.131...v1.0.132 )
Updates `github.com/projectdiscovery/gologger` from 1.1.60 to 1.1.61
- [Release notes](https://github.com/projectdiscovery/gologger/releases )
- [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.60...v1.1.61 )
Updates `github.com/projectdiscovery/networkpolicy` from 0.1.28 to 0.1.29
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases )
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.28...v0.1.29 )
Updates `github.com/projectdiscovery/tlsx` from 1.2.1 to 1.2.2
- [Release notes](https://github.com/projectdiscovery/tlsx/releases )
- [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml )
- [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.2.1...v1.2.2 )
Updates `github.com/projectdiscovery/useragent` from 0.0.102 to 0.0.103
- [Release notes](https://github.com/projectdiscovery/useragent/releases )
- [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.102...v0.0.103 )
Updates `github.com/projectdiscovery/utils` from 0.6.1 to 0.7.1
- [Release notes](https://github.com/projectdiscovery/utils/releases )
- [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md )
- [Commits](https://github.com/projectdiscovery/utils/compare/v0.6.1...v0.7.1 )
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.55 to 0.2.56
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases )
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.55...v0.2.56 )
Updates `github.com/projectdiscovery/cdncheck` from 1.2.10 to 1.2.11
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases )
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml )
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.10...v1.2.11 )
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
dependency-version: 0.4.17
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/hmap
dependency-version: 0.0.96
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryabledns
dependency-version: 1.0.109
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.132
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/gologger
dependency-version: 1.1.61
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
dependency-version: 0.1.29
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/tlsx
dependency-version: 1.2.2
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/useragent
dependency-version: 0.0.103
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/utils
dependency-version: 0.7.1
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.56
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.2.11
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com >
* chore: update utils.httputil calls
Signed-off-by: Dwi Siswanto <git@dw1.io >
* chore(deps): bump github.com/projectdiscovery/utils => v0.7.3
Signed-off-by: Dwi Siswanto <git@dw1.io >
---------
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: Dwi Siswanto <git@dw1.io >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dwi Siswanto <git@dw1.io >
* chore(deps): bump the modules group with 11 updates
Bumps the modules group with 11 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer ) | `0.4.17` | `0.4.18` |
| [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap ) | `0.0.96` | `0.0.97` |
| [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns ) | `1.0.109` | `1.0.110` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go ) | `1.0.132` | `1.0.133` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl ) | `0.8.5` | `0.8.6` |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger ) | `1.1.61` | `1.1.62` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy ) | `0.1.29` | `0.1.30` |
| [github.com/projectdiscovery/uncover](https://github.com/projectdiscovery/uncover ) | `1.1.0` | `1.2.0` |
| [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent ) | `0.0.103` | `0.0.104` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo ) | `0.2.56` | `0.2.57` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck ) | `1.2.11` | `1.2.12` |
Updates `github.com/projectdiscovery/fastdialer` from 0.4.17 to 0.4.18
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases )
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.17...v0.4.18 )
Updates `github.com/projectdiscovery/hmap` from 0.0.96 to 0.0.97
- [Release notes](https://github.com/projectdiscovery/hmap/releases )
- [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.96...v0.0.97 )
Updates `github.com/projectdiscovery/retryabledns` from 1.0.109 to 1.0.110
- [Release notes](https://github.com/projectdiscovery/retryabledns/releases )
- [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.109...v1.0.110 )
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.132 to 1.0.133
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases )
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.132...v1.0.133 )
Updates `github.com/projectdiscovery/dsl` from 0.8.5 to 0.8.6
- [Release notes](https://github.com/projectdiscovery/dsl/releases )
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.5...v0.8.6 )
Updates `github.com/projectdiscovery/gologger` from 1.1.61 to 1.1.62
- [Release notes](https://github.com/projectdiscovery/gologger/releases )
- [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.61...v1.1.62 )
Updates `github.com/projectdiscovery/networkpolicy` from 0.1.29 to 0.1.30
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases )
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.29...v0.1.30 )
Updates `github.com/projectdiscovery/uncover` from 1.1.0 to 1.2.0
- [Release notes](https://github.com/projectdiscovery/uncover/releases )
- [Commits](https://github.com/projectdiscovery/uncover/compare/v1.1.0...v1.2.0 )
Updates `github.com/projectdiscovery/useragent` from 0.0.103 to 0.0.104
- [Release notes](https://github.com/projectdiscovery/useragent/releases )
- [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.103...v0.0.104 )
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.56 to 0.2.57
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases )
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.56...v0.2.57 )
Updates `github.com/projectdiscovery/cdncheck` from 1.2.11 to 1.2.12
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases )
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.11...v1.2.12 )
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
dependency-version: 0.4.18
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/hmap
dependency-version: 0.0.97
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryabledns
dependency-version: 1.0.110
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.133
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
dependency-version: 0.8.6
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/gologger
dependency-version: 1.1.62
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
dependency-version: 0.1.30
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/uncover
dependency-version: 1.2.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/useragent
dependency-version: 0.0.104
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.57
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.2.12
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com >
* feat(loader): implement persistent metadata cache (#6630 )
* feat(loader): implement persistent metadata cache
for template filtering optimization.
Introduce a new template metadata indexing system
with persistent caching to dramatically improve
template loading perf when filters are applied.
The implementation adds a new index pkg that
caches lightweight template metadata (ID, tags,
authors, severity, .etc) and enables filtering
templates before expensive YAML parsing occurs.
The index uses an in-memory LRU cache backed by
`otter` pkg for efficient memory management with
adaptive sizing based on entry weight, defaulting
to approx. 40MB for 50K templates.
Metadata is persisted to disk using gob encoding
at "~/.cache/nuclei/index.gob" with atomic writes
to prevent corruption. The cache automatically
invalidates stale entries using `ModTime` to
detect file modifications, ensuring metadata
freshness w/o manual intervention.
Filtering has been refactored from the previous
`TagFilter` and `PathFilter` approach into a
unified `index.Filter` type that handles all basic
filtering ops including severity, authors, tags,
template IDs with wildcard support, protocol
types, and path-based inclusion and exclusion. The
filter implements OR logic within each field type
and AND logic across different field types, with
exclusion filters taking precedence over inclusion
filters and forced inclusion via
`IncludeTemplates` and `IncludeTags` overriding
exclusions.
The `loader` integration creates an index filter
from store configuration via `buildIndexFilter`
and manages the cache lifecycle through
`loadTemplatesIndex` and `saveTemplatesIndex`
methods. When `LoadTemplatesOnlyMetadata` or
`LoadTemplatesWithTags` is called, the system
first checks the metadata cache for each template
path. If cached metadata exists and passes
validation, the filter is applied directly against
the metadata without parsing. Only templates
matching the filter criteria proceed to full YAML
parsing, resulting in significant performance
gains.
Advanced filtering via "-tc" flag
(`IncludeConditions`) still requires template
parsing as these are expression-based filters that
cannot be evaluated from metadata alone. The
`TagFilter` has been simplified to handle only
`IncludeConditions` while all other filtering ops
are delegated to the index-based filtering system.
Cache management is fully automatic with no user
configuration required. The cache gracefully
handles errors by logging warnings & falling back
to normal op w/o caching. Cache files use schema
versioning to invalidate incompatible cache
formats across nuclei updates (well, specifically
`Index` and `Metadata` changes).
This optimization particularly benefits repeated
scans with the same filters, CI/CD pipelines
running nuclei regularly, development and testing
workflows with frequent template loading, and any
scenario with large template collections where
filtering would exclude most templates.
* test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs
Signed-off-by: Dwi Siswanto <git@dw1.io >
* ci: cache nuclei-templates index
Signed-off-by: Dwi Siswanto <git@dw1.io >
* chore(index): satisfy lints
Signed-off-by: Dwi Siswanto <git@dw1.io >
* fix(index): correct metadata filter logic
for proper template matching.
The `filter.matchesIncludes()` was using OR logic
across different filter types, causing incorrect
template matching. Additionally, ID matching was
case-sensitive, failing to match patterns like
'CVE-2021-*'.
The filter now correctly implements: (author1 OR
author2) AND (tag1 OR tag2) AND (severity1 OR
severity2) - using OR within each filter type and
AND across different types.
Signed-off-by: Dwi Siswanto <git@dw1.io >
* test(index): resolve test timing issue
in CI environments.
Some test was failing in CI due to filesystem
timestamp resolution limitations. On filesystems
with 1s ModTime granularity (common in CI),
modifying a file immediately after capturing its
timestamp resulted in identical ModTime values,
causing IsValid() to incorrectly return true.
Signed-off-by: Dwi Siswanto <git@dw1.io >
* ci: cache nuclei with composite action
Signed-off-by: Dwi Siswanto <git@dw1.io >
* fix(index): file locking issue on Windows
during cache save/load.
Explicitly close file handles before performing
rename/remove ops in `Save` and `Load` methods.
* In `Save`, close temp file before rename.
* In `Load`, close file before remove during error
handling/version mismatch.
Signed-off-by: Dwi Siswanto <git@dw1.io >
* test(index): flaky index tests on Windows
Fix path separator mismatch in `TestCacheSize`
and `TestCachePersistenceWithLargeDataset` by
using `filepath.Join` consistently instead of
hardcoded forward slashes.
Signed-off-by: Dwi Siswanto <git@dw1.io >
* test(cmd): init logger to prevent nil pointer deref
The integration tests were panicking with a nil
pointer dereference in `pkg/catalog/loader`
because the logger was not init'ed.
When `store.saveMetadataIndexOnce` attempted to
log the result of the metadata cache op, it
dereferenced the nil logger, causing a crash.
Signed-off-by: Dwi Siswanto <git@dw1.io >
* fix(loader): resolve include/exclude paths
for metadata cache filter.
The `indexFilter` was previously init'ed using raw
relative paths from the config for
`IncludeTemplates` and `ExcludeTemplates`.
But the persistent metadata cache stores templates
using their absolute paths. This mismatch caused
the `matchesPath` check to fail, leading to
templates being incorrectly excluded even when
explicitly included via flags
(e.g., "-include-templates
loader/excluded-template.yaml").
This commit updates `buildIndexFilter` to resolve
these paths to their absolute versions using
`store.config.Catalog.GetTemplatesPath` before
creating the filter, ensuring consistent path
matching against the metadata cache.
Signed-off-by: Dwi Siswanto <git@dw1.io >
* feat(index): adds `NewMetadataFromTemplate` func
Signed-off-by: Dwi Siswanto <git@dw1.io >
* refactor(index): return metadata when `(*Index).cache` is nil
Signed-off-by: Dwi Siswanto <git@dw1.io >
* refactor(loader): restore pre‑index behavior semantics
Signed-off-by: Dwi Siswanto <git@dw1.io >
---------
Signed-off-by: Dwi Siswanto <git@dw1.io >
* chore: bump version
Signed-off-by: Dwi Siswanto <git@dw1.io >
---------
Signed-off-by: Dwi Siswanto <git@dw1.io >
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: ledigang <shuangcui@msn.com >
Co-authored-by: pussycat0x <65701233+pussycat0x@users.noreply.github.com >
Co-authored-by: Mzack9999 <mzack9999@protonmail.com >
Co-authored-by: tvroi <roy.oswaldha@traveloka.com >
Co-authored-by: Niek den Breeje <n.denbreeje@guardian360.nl >
Co-authored-by: circleous <circleousdev@gmail.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ice3man <nizamulrana@gmail.com >
Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com >
Co-authored-by: ledigang <shuangcui@msn.com >
Co-authored-by: Doğan Can Bakır <dogancanbakir@protonmail.com >
2025-12-05 03:37:18 +07:00
Nakul Bharti
c4fa2c74c1
cache, goroutine and unbounded workers management ( #6420 )
...
* Enhance matcher compilation with caching for regex and DSL expressions to improve performance. Update template parsing to conditionally retain raw templates based on size constraints.
* Implement caching for regex and DSL expressions in extractors and matchers to enhance performance. Introduce a buffer pool in raw requests to reduce memory allocations. Update template cache management for improved efficiency.
* feat: improve concurrency to be bound
* refactor: replace fmt.Sprintf with fmt.Fprintf for improved performance in header handling
* feat: add regex matching tests and benchmarks for performance evaluation
* feat: add prefix check in regex extraction to optimize matching process
* feat: implement regex caching mechanism to enhance performance in extractors and matchers, along with tests and benchmarks for validation
* feat: add unit tests for template execution in the core engine, enhancing test coverage and reliability
* feat: enhance error handling in template execution and improve regex caching logic for better performance
* Implement caching for regex and DSL expressions in the cache package, replacing previous sync.Map usage. Add unit tests for cache functionality, including eviction by capacity and retrieval of cached items. Update extractors and matchers to utilize the new cache system for improved performance and memory efficiency.
* Add tests for SetCapacities in cache package to ensure cache behavior on capacity changes
- Implemented TestSetCapacities_NoRebuildOnZero to verify that setting capacities to zero does not clear existing caches.
- Added TestSetCapacities_BeforeFirstUse to confirm that initial cache settings are respected and not overridden by subsequent capacity changes.
* Refactor matchers and update load test generator to use io package
- Removed maxRegexScanBytes constant from match.go.
- Replaced ioutil with io package in load_test.go for NopCloser usage.
- Restored TestValidate_AllowsInlineMultiline in load_test.go to ensure inline validation functionality.
* Add cancellation support in template execution and enhance test coverage
- Updated executeTemplateWithTargets to respect context cancellation.
- Introduced fakeTargetProvider and slowExecuter for testing.
- Added Test_executeTemplateWithTargets_RespectsCancellation to validate cancellation behavior during template execution.
2025-09-15 23:48:02 +05:30
Tarun Koyalwar
19247ae74b
Path-Based Fuzzing SQL fix ( #6400 )
...
* setup claude
* migrate to using errkit
* fix unused imports + lint errors
* update settings.json
* fix url encoding issue
* fix lint error
* fix the path fuzzing component
* fix lint error
2025-08-25 13:36:58 +05:30
Sandeep Singh
b4644af80a
Lint + test fixes after utils dep update ( #6393 )
...
* fix: remove undefined errorutil.ShowStackTrace
* feat: add make lint support and integrate with test
* refactor: migrate errorutil to errkit across codebase
- Replace deprecated errorutil with modern errkit
- Convert error declarations from var to func for better compatibility
- Fix all SA1019 deprecation warnings
- Maintain error chain support and stack traces
* fix: improve DNS test reliability using Google DNS
- Configure test to use Google DNS (8.8.8.8) for stability
- Fix nil pointer issue in DNS client initialization
- Keep production defaults unchanged
* fixing logic
* removing unwanted branches in makefile
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com >
2025-08-20 05:28:23 +05:30
gopherorg
1079498182
refactor: use maps.Copy for cleaner map handling ( #6283 )
...
Signed-off-by: gopherorg <gopherworld@icloud.com >
2025-07-12 02:50:47 +05:30
HD Moore
f26996cb89
Remove singletons from Nuclei engine (continuation of #6210 ) ( #6296 )
...
* introducing execution id
* wip
* .
* adding separate execution context id
* lint
* vet
* fixing pg dialers
* test ignore
* fixing loader FD limit
* test
* fd fix
* wip: remove CloseProcesses() from dev merge
* wip: fix merge issue
* protocolstate: stop memguarding on last dialer delete
* avoid data race in dialers.RawHTTPClient
* use shared logger and avoid race conditions
* use shared logger and avoid race conditions
* go mod
* patch executionId into compiled template cache
* clean up comment in Parse
* go mod update
* bump echarts
* address merge issues
* fix use of gologger
* switch cmd/nuclei to options.Logger
* address merge issues with go.mod
* go vet: address copy of lock with new Copy function
* fixing tests
* disable speed control
* fix nil ExecuterOptions
* removing deprecated code
* fixing result print
* default logger
* cli default logger
* filter warning from results
* fix performance test
* hardcoding path
* disable upload
* refactor(runner): uses `Warning` instead of `Print` for `pdcpUploadErrMsg`
Signed-off-by: Dwi Siswanto <git@dw1.io >
* Revert "disable upload"
This reverts commit 114fbe6663cf12ca800egit@dw1.io >
Co-authored-by: Mzack9999 <mzack9999@protonmail.com >
Co-authored-by: Dwi Siswanto <git@dw1.io >
Co-authored-by: Dwi Siswanto <25837540+dwisiswant0@users.noreply.github.com >
2025-07-10 01:17:26 +05:30
Tarun Koyalwar
2b729e4037
fix context leak in flow ( #6282 )
...
* fix context leak in flow
* handle sizedwaitpool when not reused
2025-06-30 16:43:00 +07:00
Doğan Can Bakır
1e08d29e50
fix unresolved interactsh-url for js templates
2025-03-06 15:52:12 +03:00
Dwi Siswanto
2c832f5590
refactor(vardump): use godump lib ( #5676 )
...
* refactor(vardump): use `godump` lib
also increate limit char to `255`.
Signed-off-by: Dwi Siswanto <git@dw1.io >
* feat(vardump): add global var `Limit`
Signed-off-by: Dwi Siswanto <git@dw1.io >
* chore(protocols): rm newline
Signed-off-by: Dwi Siswanto <git@dw1.io >
* feat(types): add `VarDumpLimit` option
Signed-off-by: Dwi Siswanto <git@dw1.io >
* test(vardump): add test cases
Signed-off-by: Dwi Siswanto <git@dw1.io >
* chore: tidy up mod
Signed-off-by: Dwi Siswanto <git@dw1.io >
---------
Signed-off-by: Dwi Siswanto <git@dw1.io >
2024-10-14 19:31:36 +05:30
Ramana Reddy
3d2f31a56f
fix missing template_url for pd signed templates when executed from custom path ( #5644 )
2024-09-19 18:58:20 +05:30
Tarun Koyalwar
2418319df4
js: generate matcher-status event ( #5450 )
...
* js: generate matcher-status event
* isPortOpen: use fastdialer instance
* update sdk unit test
* add docs :)
2024-07-27 02:46:34 +05:30
Dogan Can Bakir
f080d614c3
introduce timeouts config in types.Options ( #5228 )
...
* introduce timeout variants
* update instances and add codeexectimeout
* fix test
* default to 10s
* minor
* make timeouts pluggable and rename
* remove residual code
---------
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io >
2024-07-15 15:57:15 +05:30
Tarun Koyalwar
23bd0336fb
multiple bug fixes + performance improvements ( #5148 )
...
* prototype errkit
* complete errkit implementation
* add cause to all timeouts
* fix request timeout annotation @timeout
* increase responseHeaderTimeout to 8 for stability
* rawhttp error related improvements
* feat: add port status caching
* add port status caching to http
* migrate to new utils/errkit
* remote dialinterface + error cause
* debug dir support using .gitignore debug-*
* make nuclei easy to debug
* debug dir update .gitignore
* temp change (to revert)
* Revert "temp change (to revert)"
This reverts commit d3131f7777sandeep@projectdiscovery.io >
2024-05-25 00:29:04 +05:30
Tarun Koyalwar
3e54ca54b0
feat: fix utils and add goroutine leak unit tests ( #5112 )
...
* feat: fixed leak
* add go leak unit test in sdk
* added goleak unit tests
* bugfix: add random user agents to fuzzing requests
* misc
* misc
* fix lint + use utils pr + misc
* fix ratelimit memleak in sdk
* close protocolstate shared resources in nuclei sdk/lib
* add missing close references
* ignore read/write loop of intransit connections
* close unnecessary idle conns
* add ignore method
* using fixed utils
* dep update
---------
Co-authored-by: Ice3man <nizamulrana@gmail.com >
Co-authored-by: mzack <marco.rivoli.nvh@gmail.com >
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com >
2024-05-01 00:28:11 +05:30
Ice3man
0b82e8b7aa
feat: added support for context cancellation to engine ( #5096 )
...
* feat: added support for context cancellation to engine
* misc
* feat: added contexts everywhere
* misc
* misc
* use granular http timeouts and increase http timeout to 30s using multiplier
* track response header timeout in mhe
* update responseHeaderTimeout to 5sec
* skip failing windows test
---------
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io >
2024-04-25 15:37:56 +05:30
mzack
7e363984b2
Merge branch 'dev' into feat-3072-init-adaptive-speed
2024-04-09 15:19:51 +02:00
Tarun Koyalwar
375d1ddcde
fix missing port in javascript result ( #5023 )
...
* add ip support in js output
* js: if dialed ip is missing resolve and get first ip
* ssl: fix incorrect port in output
2024-04-09 02:09:44 +05:30
mzack
af7450737a
making payload concurrency dynamic via direct int change
2024-04-03 23:06:08 +02:00
Mzack9999
a8d1393e96
init- using resizable components
2024-04-03 17:50:57 +02:00
Tarun Koyalwar
8a2ff17ad8
allow specifying self-contained at http request level ( #4812 )
...
* allow specifying self-contained at requestlevel
* fix IsSMTP js example
* update smtp + fix examples
* update smtp error message
* add code reference in js protocol
* update js docs
* remove debug stmt
2024-03-01 16:38:56 +05:30
Tarun Koyalwar
cc732875cd
javascript: pooling and reuse with export functions + misc updates ( #4709 )
...
* js hotfix: wrap javascript source in anon functions
* mysql module improvements
* misc mysql bugs
* js vm pooling: soft deprecation + incentivised pooling
* misc updates
* disable interactsh failed test
* disable interactsh.yaml integration test on win & mac
2024-02-02 02:22:04 +05:30
Tarun Koyalwar
ead58f4ab9
implicit thread count when not specified in payloads + threads support in dns,network ( #4715 )
...
* default threads + add threads support in dns payloads
* add threads support in network protocol
* add optional callback to override threadSetter
* fix broken fuzz integration tests
2024-02-02 02:05:30 +05:30
Tarun Koyalwar
5bd9d9ee68
memory leak fixes and optimizations ( #4680 )
...
* feat http response memory optimization + reuse buffers
* update nuclei version
* feat: reuse js vm's and compile to programs
* fix failing http integration test
* remove dead code + add -jsc
* feat reuse js vms in pool with concurrency
* update comments as per review
* bug fix+ update interactsh test to look for dns interaction
* try enabling all interactsh integration tests
---------
Co-authored-by: mzack <marco.rivoli.nvh@gmail.com >
2024-01-31 01:59:49 +05:30
Tarun Koyalwar
c7c35ffb94
fix multiple mem leaks + optimizations ( #4630 )
...
* fix mem leak
* bump version tag
* http: add global resp body read limit of 4MB
* skip creating templateCtx in normal templates
* fix mem leak via retryablehttp , fastdialer
* go mod tidy
* remove unused var
* dep update
---------
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com >
2024-01-18 05:53:42 +05:30
Tarun Koyalwar
a677fca192
misc improvements in js protocol execution ( #4643 )
...
* js protocol timeout using -timeout flag
* fix zgrab smb hang
* fix lint error
* custom timeout field in js protocol
* minor update: bound checking
* add 6 * -timeout in code protocol by default
2024-01-18 04:39:15 +05:30
Tarun Koyalwar
6e969cbd3c
add additional json fields: port,ip,scheme,url ( #4417 )
...
* add additional json fields: port,ip,scheme,url
* include host field in case of ip input
2023-11-28 14:26:23 +05:30
Dogan Can Bakir
ce5df9cc02
introduce scan context ( #4373 )
...
* introduce scan context
* minor
* add joined errors to resultevents
* change `executor` funcs' signature
* fix tests
* join errors in `LogError` func
* change func signature
* add guard
2023-11-28 00:24:45 +05:30
Dogan Can Bakir
7c2db9c394
introduce template-encoded field ( #4315 )
...
* introduce `template-encoded` field
* remove IsCustomTemplate func
* refactor and move encoding to `MakeResultEventItem` func
* encode template in case of no results were found
* commit to last commit
* don't encode templates when`-ms` is used
2023-11-11 04:42:27 +05:30
Tarun Koyalwar
595ba8e3a5
bug fixes in js , network protocol and flow ( #4313 )
...
* fix net read
* only return N bytes if extra available
* use ConnReadN from readerutil
* add integration test
* print unsigned warning in stderr
* fix js protocol in flow #4318
* fix integration test: url encoding issue
* fix network protocol issue + integration tests
* multiple improvements to integration test
* replace all conn.Read() from tests
* disable network-basic.yaml in windows
* disable code protocol in win CI
* fix bitwise login ps1-snippet.yaml
* hide previous matcher events in flow
* remove dead code+ update integration tests
---------
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io >
2023-11-02 13:33:40 +05:30
Tarun Koyalwar
dc44105baf
nuclei v3 : misc updates ( #4247 )
...
* use parsed options while signing
* update project layout to v3
* fix .gitignore
* remove example template
* misc updates
* bump tlsx version
* hide template sig warning with env
* js: retain value while using log
* fix nil pointer derefernce
* misc doc update
---------
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com >
2023-10-17 17:44:13 +05:30
