The `race` condition directive was broken due to
a strict dependency on `threads > 0` for parallel
execution, causing templates with `race` directive
enabled but no explicit threads to fall back to
seq execution.
This regression was introduced in v3.2.0 (#4868),
which restricted parallel execution to only when
`payloads` were present.
Fixes#5713 to allow race conditions even w/o
explicit `payloads`, and add a default thread
count when race is enabled but threads is 0.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* adding min auth support
* adding unauth list modules + auth list files in module
* example
* adding rsync test
* bump go.mod
---------
Co-authored-by: Dwi Siswanto <git@dw1.io>
`hasMatchers` was not nil-safe when iterating over
the slice of operators. Check if the operator is
nil before accessing
`*operators.Operators.Matchers` to prevent a panic
when a protocol implementation returns a slice
containing a nil element.
This can happen when a request has no local
matchers/extractors but is processed in a flow
where global matchers are present.
Fixes#6738.
Signed-off-by: Dwi Siswanto <git@dw1.io>
The `connectWithDSN` func used `db.Exec()` which
implicitly uses `context.Background()`[1]. This
caused the registered "nucleitcp" dialer
callback to receive a ctx missing the
`executionId`, leading to a panic during type
assertion.
Refactor `connectWithDSN` to accept `executionId`
explicitly and use it to create a `context` for
`db.PingContext()` (yeah, instead of `db.Exec()`).
And, add a defensive check in the dialer callback
to handle nil values gracefully.
Fixes#6733 regression introduced in #6296.
[1]: "Exec uses `context.Background` internally" -
https://pkg.go.dev/database/sql#DB.Exec.
Signed-off-by: Dwi Siswanto <git@dw1.io>
Make sure postgres Exec/ExecContext are invoked with the correct
argument order, preventing context from being passed as the query.
* fixing pg syntax
* adding test
`MergeMaps` accounts for 11.41% of allocs (13.8
GB) in clusterbomb mode. With 1,305 combinations
per target, this function is called millions of
times in the hot path.
RCA:
* Request generator calls `MergeMaps` with single
arg on every payload combination, incurring
variadic overhead.
* Build request merges same maps multiple times
per request.
* `BuildPayloadFromOptions` recomputes static CLI
options on every call.
* Variables calls `MergeMaps` $$2×N$$ times per
variable evaluation (once in loop, once in
`evaluateVariableValue`)
Changes:
Core optimizations in maps.go:
* Pre-size merged map to avoid rehashing (30-40%
reduction)
* Add `CopyMap` for efficient single-map copy
without variadic overhead.
* Add `MergeMapsInto` for in-place mutation when
caller owns destination.
Hot path fixes:
* Replace `MergeMaps(r.currentPayloads)` with
`CopyMap(r.currentPayloads)` to eliminates
allocation on every combination iteration.
* Pre-allocate combined map once, extend in-place
during `ForEach` loop instead of creating new
map per variable (eliminates $$2×N$$ allocations
per request).
Caching with concurrency safety:
* Cache `BuildPayloadFromOptions` computation in
`sync.Map` keyed by `types.Options` ptr, but
return copy to prevent concurrent modification.
* Cost: shallow copy of ~10-20 entries vs. full
merge of vars + env (85-90% savings in typical
case)
* Clear cache in `closeInternal()` to prevent
memory leaks when SDK instances are created or
destroyed.
Estimated impact: 40-60% reduction in `MergeMaps`
allocations (5.5-8.3 GB savings from original
13.8 GB). Safe for concurrent execution and SDK
usage with multiple instances.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat: add site-url to optionally provide jira server URL for oauth
* chore(cmd): add `site-url` config option
Adds optional `site-url` field to JIRA issue
tracker configuration for specifying browsable URL
when it differs from the API endpoint. This is
particularly useful for OAuth-based JIRA Cloud
integrations where `issue.Self` contains
"api.atlassian.com" instead of the user-facing
domain.
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Dwi Siswanto <git@dw1.io>
This patch enables TLS session resumption by
setting a shared LRU session cache
(`ClientSessionCache`) in all HTTP client TLS
configs. This reduces handshake overhead and CPU
usage for repeated conns to the same host,
improving throughput and efficiency in
clusterbomb/pitchfork modes.
This applied to HTTP-request-based and headless-
request-based protocols.
No runtime/compatibility impact.
Signed-off-by: Dwi Siswanto <git@dw1.io>
with configurable limits
This patch fixes duplicate issue detection for
GitLab trackers by implementing paginated search
with configurable page size and max pages. Adds
`duplicate-issue-page-size` and
`duplicate-issue-max-pages` options to the config.
Fixes#6711.
Signed-off-by: Dwi Siswanto <git@dw1.io>
Restore backwards compat for JavaScript protocol
templates that omit the `Port` argument.
Regression was introduced in f4f2e9f2, which
removed the fallback for empty `Port` in
`(*Request).ExecuteWithResults`, causing templates
without `Port` to be silently skipped.
Now, if no `Port` is specified, the engine
executes the JavaScript block using the target
URL's port.
Fixes#6708.
Signed-off-by: Dwi Siswanto <git@dw1.io>
Continue the fix from #6666 by converting
remaining direct Body assignments to use setter
methods:
* pkg/fuzz/component/body.go:139: use
`SetBodyReader()` in transfer-encoding path.
* pkg/protocols/http/request.go:694: use
`SetBodyString()` in fuzz component `Rebuild()`.
Fixes#6692.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(http): pass `dynamicValues` to `EvaluateWithInteractsh`
When `LazyEval` is true (triggered by `variables`
containing `BaseURL`, `Hostname`,
`interactsh-url`, etc.), variable expressions are not
eval'ed during YAML parsing & remain as raw exprs
like "{{rand_base(5)}}".
At request build time, `EvaluateWithInteractsh()`
checks if a variable already has a value in the
passed map before re-evaluating its expression.
But, `dynamicValues` (which contains the template
context with previously eval'ed values) was not
being passed, causing exprs like `rand_*` to be
re-evaluated on each request, producing different
values.
Fixes#6684 by including `dynamicValues` in the
map passed to `EvaluateWithInteractsh()`, so
variables evaluated in earlier requests retain
their values in subsequent requests.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(http): rm early eval in `(*Request).ExecuteWithResults()`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds variables-threads-previous integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds constants-with-threads integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds race-with-variables integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
The `(*Page).HistoryData` was being read w/o
holding the mutex lock after
`(*Page).ExecuteActions()` returns, while the
background hijack goroutine could still be writing
to it via `(*Page).addToHistory()`.
Copy the first history item by value while holding
RLock to avoid racing with concurrent append ops.
Fixes#6686.
Signed-off-by: Dwi Siswanto <git@dw1.io>
Prev, `FullResponseString()`, `BodyString()`, and
`HeadersString()` were called multiple times per
HTTP response iteration, each call allocating a
new string copy of the response data.
For a 10MB response, this resulted in ~60MB of
redundant string allocs/response (6 calls x 10MB).
Cache the string representations once per `Fill()`
cycle and reuse them throughout the response
processing loop. This reduces allocs from 6 to 3
per response, cutting memory usage by ~50% for
response string handling.
Profiling showed these functions accounting for
~89% of heap allocs (5.7GB out of 6.17GB) during
large scans.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(http): lost request body on retries & redirects
Updates the HTTP protocol to use
`(*retryablehttp.Request).SetBodyString` instead
of direct `Body` assignment.
This fixes#6665 where the request body was
dropped during retries or 307/308 redirects
because `GetBody` was not being populated.
Thanks to @zzyjsj for reporting the bug in the
upstream dependency and the hints!
Signed-off-by: Dwi Siswanto <git@dw1.io>
* empty: add co-author
Co-authored-by: zzy <zzyjsj@users.noreply.github.com>
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: zzy <zzyjsj@users.noreply.github.com>
Prev, the template exclusion logic checked if the
full file path contained any of the known
miscellaneous directory names (e.g., helpers,
.git). This caused false positives when valid
templates were stored in paths where a parent
directory matched one of these names (e.g.,
/path/to/somewhere/that/has/helpers/dir/name/).
This commit introduces `IsTemplateWithRoot`, which
checks for excluded directories relative to a
provided root directory. It splits the relative
path into components.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(lib): segfault when init engine with `EnableHeadlessWithOpts`
The panic was caused by attempting to log a
sandbox warning before the logger was initialized.
RCA:
* SDK option funcs were exec'd before logger init.
* `EnableHeadlessWithOpts()` attempted to create
browser instance & log warnings during the
config phase.
* `Logger` was only init'd later in `init()`
phase.
* This caused nil pointer dereference when
`MustDisableSandbox()` returned true (root on
Linux/Unix or Windows).
Changes:
* Init `Logger` in `types.DefaultOptions()` to
ensure it's always available before any option
functions execute.
* Init `Logger` field in both
`NewNucleiEngineCtx()` and
`NewThreadSafeNucleiEngineCtx()` from
`defaultOptions.Logger`.
* Move browser instance creation from
`EnableHeadlessWithOpts()` to the `init()` phase
where `Logger` is guaranteed to be available.
* Simplify logger sync logic in `init()` to only
update if changed by `WithLogger` option.
* Add test case to verify headless initialization
works without panic.
The fix maintains backward compatibility while
make sure the logger is always available when
needed by any SDK option function.
Fixes#6601.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* build(make): adds `-timeout 30m -count 1` GOFLAGS in `test` cmd
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Revert "fix(lib): segfault when init engine with `EnableHeadlessWithOpts`"
let see if this pass flaky test.
This reverts commit 63fcb6a1cbe7a4db7a78be766affc70eb237e57e.
* test(engine): let see if this pass flaky test
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Revert "Revert "fix(lib): segfault when init engine with `EnableHeadlessWithOpts`""
This reverts commit 62b4223803ccb1e93593e2e08e39923d76aa20b1.
* test(engine): increase `TestActionNavigate` timeout
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Revert "test(engine): let see if this pass flaky test"
This reverts commit d27cd985cff1b06aa1965ea11f8aa32f00778ab5.
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(loader): implement persistent metadata cache
for template filtering optimization.
Introduce a new template metadata indexing system
with persistent caching to dramatically improve
template loading perf when filters are applied.
The implementation adds a new index pkg that
caches lightweight template metadata (ID, tags,
authors, severity, .etc) and enables filtering
templates before expensive YAML parsing occurs.
The index uses an in-memory LRU cache backed by
`otter` pkg for efficient memory management with
adaptive sizing based on entry weight, defaulting
to approx. 40MB for 50K templates.
Metadata is persisted to disk using gob encoding
at "~/.cache/nuclei/index.gob" with atomic writes
to prevent corruption. The cache automatically
invalidates stale entries using `ModTime` to
detect file modifications, ensuring metadata
freshness w/o manual intervention.
Filtering has been refactored from the previous
`TagFilter` and `PathFilter` approach into a
unified `index.Filter` type that handles all basic
filtering ops including severity, authors, tags,
template IDs with wildcard support, protocol
types, and path-based inclusion and exclusion. The
filter implements OR logic within each field type
and AND logic across different field types, with
exclusion filters taking precedence over inclusion
filters and forced inclusion via
`IncludeTemplates` and `IncludeTags` overriding
exclusions.
The `loader` integration creates an index filter
from store configuration via `buildIndexFilter`
and manages the cache lifecycle through
`loadTemplatesIndex` and `saveTemplatesIndex`
methods. When `LoadTemplatesOnlyMetadata` or
`LoadTemplatesWithTags` is called, the system
first checks the metadata cache for each template
path. If cached metadata exists and passes
validation, the filter is applied directly against
the metadata without parsing. Only templates
matching the filter criteria proceed to full YAML
parsing, resulting in significant performance
gains.
Advanced filtering via "-tc" flag
(`IncludeConditions`) still requires template
parsing as these are expression-based filters that
cannot be evaluated from metadata alone. The
`TagFilter` has been simplified to handle only
`IncludeConditions` while all other filtering ops
are delegated to the index-based filtering system.
Cache management is fully automatic with no user
configuration required. The cache gracefully
handles errors by logging warnings & falling back
to normal op w/o caching. Cache files use schema
versioning to invalidate incompatible cache
formats across nuclei updates (well, specifically
`Index` and `Metadata` changes).
This optimization particularly benefits repeated
scans with the same filters, CI/CD pipelines
running nuclei regularly, development and testing
workflows with frequent template loading, and any
scenario with large template collections where
filtering would exclude most templates.
* test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs
Signed-off-by: Dwi Siswanto <git@dw1.io>
* ci: cache nuclei-templates index
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(index): satisfy lints
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(index): correct metadata filter logic
for proper template matching.
The `filter.matchesIncludes()` was using OR logic
across different filter types, causing incorrect
template matching. Additionally, ID matching was
case-sensitive, failing to match patterns like
'CVE-2021-*'.
The filter now correctly implements: (author1 OR
author2) AND (tag1 OR tag2) AND (severity1 OR
severity2) - using OR within each filter type and
AND across different types.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(index): resolve test timing issue
in CI environments.
Some test was failing in CI due to filesystem
timestamp resolution limitations. On filesystems
with 1s ModTime granularity (common in CI),
modifying a file immediately after capturing its
timestamp resulted in identical ModTime values,
causing IsValid() to incorrectly return true.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* ci: cache nuclei with composite action
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(index): file locking issue on Windows
during cache save/load.
Explicitly close file handles before performing
rename/remove ops in `Save` and `Load` methods.
* In `Save`, close temp file before rename.
* In `Load`, close file before remove during error
handling/version mismatch.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(index): flaky index tests on Windows
Fix path separator mismatch in `TestCacheSize`
and `TestCachePersistenceWithLargeDataset` by
using `filepath.Join` consistently instead of
hardcoded forward slashes.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(cmd): init logger to prevent nil pointer deref
The integration tests were panicking with a nil
pointer dereference in `pkg/catalog/loader`
because the logger was not init'ed.
When `store.saveMetadataIndexOnce` attempted to
log the result of the metadata cache op, it
dereferenced the nil logger, causing a crash.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(loader): resolve include/exclude paths
for metadata cache filter.
The `indexFilter` was previously init'ed using raw
relative paths from the config for
`IncludeTemplates` and `ExcludeTemplates`.
But the persistent metadata cache stores templates
using their absolute paths. This mismatch caused
the `matchesPath` check to fail, leading to
templates being incorrectly excluded even when
explicitly included via flags
(e.g., "-include-templates
loader/excluded-template.yaml").
This commit updates `buildIndexFilter` to resolve
these paths to their absolute versions using
`store.config.Catalog.GetTemplatesPath` before
creating the filter, ensuring consistent path
matching against the metadata cache.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(index): adds `NewMetadataFromTemplate` func
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(index): return metadata when `(*Index).cache` is nil
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(loader): restore pre‑index behavior semantics
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
