Continue the fix from #6666 by converting
remaining direct Body assignments to use setter
methods:
* pkg/fuzz/component/body.go:139: use
`SetBodyReader()` in transfer-encoding path.
* pkg/protocols/http/request.go:694: use
`SetBodyString()` in fuzz component `Rebuild()`.
Fixes#6692.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(http): pass `dynamicValues` to `EvaluateWithInteractsh`
When `LazyEval` is true (triggered by `variables`
containing `BaseURL`, `Hostname`,
`interactsh-url`, etc.), variable expressions are not
eval'ed during YAML parsing & remain as raw exprs
like "{{rand_base(5)}}".
At request build time, `EvaluateWithInteractsh()`
checks if a variable already has a value in the
passed map before re-evaluating its expression.
But, `dynamicValues` (which contains the template
context with previously eval'ed values) was not
being passed, causing exprs like `rand_*` to be
re-evaluated on each request, producing different
values.
Fixes#6684 by including `dynamicValues` in the
map passed to `EvaluateWithInteractsh()`, so
variables evaluated in earlier requests retain
their values in subsequent requests.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(http): rm early eval in `(*Request).ExecuteWithResults()`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds variables-threads-previous integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds constants-with-threads integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds race-with-variables integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
The `(*Page).HistoryData` was being read w/o
holding the mutex lock after
`(*Page).ExecuteActions()` returns, while the
background hijack goroutine could still be writing
to it via `(*Page).addToHistory()`.
Copy the first history item by value while holding
RLock to avoid racing with concurrent append ops.
Fixes#6686.
Signed-off-by: Dwi Siswanto <git@dw1.io>
Prev, `FullResponseString()`, `BodyString()`, and
`HeadersString()` were called multiple times per
HTTP response iteration, each call allocating a
new string copy of the response data.
For a 10MB response, this resulted in ~60MB of
redundant string allocs/response (6 calls x 10MB).
Cache the string representations once per `Fill()`
cycle and reuse them throughout the response
processing loop. This reduces allocs from 6 to 3
per response, cutting memory usage by ~50% for
response string handling.
Profiling showed these functions accounting for
~89% of heap allocs (5.7GB out of 6.17GB) during
large scans.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(http): lost request body on retries & redirects
Updates the HTTP protocol to use
`(*retryablehttp.Request).SetBodyString` instead
of direct `Body` assignment.
This fixes#6665 where the request body was
dropped during retries or 307/308 redirects
because `GetBody` was not being populated.
Thanks to @zzyjsj for reporting the bug in the
upstream dependency and the hints!
Signed-off-by: Dwi Siswanto <git@dw1.io>
* empty: add co-author
Co-authored-by: zzy <zzyjsj@users.noreply.github.com>
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: zzy <zzyjsj@users.noreply.github.com>
* fix(lib): segfault when init engine with `EnableHeadlessWithOpts`
The panic was caused by attempting to log a
sandbox warning before the logger was initialized.
RCA:
* SDK option funcs were exec'd before logger init.
* `EnableHeadlessWithOpts()` attempted to create
browser instance & log warnings during the
config phase.
* `Logger` was only init'd later in `init()`
phase.
* This caused nil pointer dereference when
`MustDisableSandbox()` returned true (root on
Linux/Unix or Windows).
Changes:
* Init `Logger` in `types.DefaultOptions()` to
ensure it's always available before any option
functions execute.
* Init `Logger` field in both
`NewNucleiEngineCtx()` and
`NewThreadSafeNucleiEngineCtx()` from
`defaultOptions.Logger`.
* Move browser instance creation from
`EnableHeadlessWithOpts()` to the `init()` phase
where `Logger` is guaranteed to be available.
* Simplify logger sync logic in `init()` to only
update if changed by `WithLogger` option.
* Add test case to verify headless initialization
works without panic.
The fix maintains backward compatibility while
make sure the logger is always available when
needed by any SDK option function.
Fixes#6601.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* build(make): adds `-timeout 30m -count 1` GOFLAGS in `test` cmd
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Revert "fix(lib): segfault when init engine with `EnableHeadlessWithOpts`"
let see if this pass flaky test.
This reverts commit 63fcb6a1cbe7a4db7a78be766affc70eb237e57e.
* test(engine): let see if this pass flaky test
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Revert "Revert "fix(lib): segfault when init engine with `EnableHeadlessWithOpts`""
This reverts commit 62b4223803ccb1e93593e2e08e39923d76aa20b1.
* test(engine): increase `TestActionNavigate` timeout
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Revert "test(engine): let see if this pass flaky test"
This reverts commit d27cd985cff1b06aa1965ea11f8aa32f00778ab5.
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(interactsh): skip DNS lookups on interactsh domains
to prevent false positives.
Prevents nuclei from resolving interactsh domains
injected in Host headers, which would cause
self-interactions to be incorrectly reported as
matches.
Changes:
* Add `GetHostname()` method to `interactsh.Client`
to expose active server domain.
* Skip CNAME DNS lookups in
`(*http.Request).addCNameIfAvailable` when
hostname matches the
`(*interactsh.Client).GetHostname`.
Fixes#6613
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(http): prevent false `interactshDomain` matches
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
across multiple layers
Fixes timeout configuration conflicts where HTTP
requests would timeout prematurely despite
configured values in `@timeout` annotations or
`-timeout` flags.
RCA:
* `retryablehttp` pkg overriding with default
30s timeout.
* Custom timeouts not propagating to
`retryablehttp` layer.
* Multiple timeout layers not sync properly.
Changes:
* Propagate custom timeouts from `@timeout`
annotations to `retryablehttp` layer.
* Adjust 5-minute maximum cap to prevent DoS via
extremely large timeouts.
* Ensure `retryableHttpOptions.Timeout` respects
`ResponseHeaderTimeout`.
* Add comprehensive tests for timeout capping
behavior.
This allows templates to override global timeout
via `@timeout` annotations while preventing abuse
thru unreasonably large timeout values.
Fixes#6560.
Signed-off-by: Dwi Siswanto <git@dw1.io>
add missing `go` keyword to anonymous funcs that
were intended to run as goroutines but were
executing synchronously instead.
Fixes#6492
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Enhance matcher compilation with caching for regex and DSL expressions to improve performance. Update template parsing to conditionally retain raw templates based on size constraints.
* Implement caching for regex and DSL expressions in extractors and matchers to enhance performance. Introduce a buffer pool in raw requests to reduce memory allocations. Update template cache management for improved efficiency.
* feat: improve concurrency to be bound
* refactor: replace fmt.Sprintf with fmt.Fprintf for improved performance in header handling
* feat: add regex matching tests and benchmarks for performance evaluation
* feat: add prefix check in regex extraction to optimize matching process
* feat: implement regex caching mechanism to enhance performance in extractors and matchers, along with tests and benchmarks for validation
* feat: add unit tests for template execution in the core engine, enhancing test coverage and reliability
* feat: enhance error handling in template execution and improve regex caching logic for better performance
* Implement caching for regex and DSL expressions in the cache package, replacing previous sync.Map usage. Add unit tests for cache functionality, including eviction by capacity and retrieval of cached items. Update extractors and matchers to utilize the new cache system for improved performance and memory efficiency.
* Add tests for SetCapacities in cache package to ensure cache behavior on capacity changes
- Implemented TestSetCapacities_NoRebuildOnZero to verify that setting capacities to zero does not clear existing caches.
- Added TestSetCapacities_BeforeFirstUse to confirm that initial cache settings are respected and not overridden by subsequent capacity changes.
* Refactor matchers and update load test generator to use io package
- Removed maxRegexScanBytes constant from match.go.
- Replaced ioutil with io package in load_test.go for NopCloser usage.
- Restored TestValidate_AllowsInlineMultiline in load_test.go to ensure inline validation functionality.
* Add cancellation support in template execution and enhance test coverage
- Updated executeTemplateWithTargets to respect context cancellation.
- Introduced fakeTargetProvider and slowExecuter for testing.
- Added Test_executeTemplateWithTargets_RespectsCancellation to validate cancellation behavior during template execution.
* fix: segfault in template caching logic
when templates had no executable requests after
option updates.
the cached templates could end up with 0 requests
and no flow execution path, resulting in a nil
engine pointer that was later derefer w/o
validation.
bug seq:
caching template (w/ valid requests) -> get cached
template -> `*ExecutorOptions.Options` copied and
modified (inconsistent) -> requests updated (with
new options -- some may be invalid, and without
recompile) -> template returned w/o validation ->
`compileProtocolRequests` -> `NewTemplateExecuter`
receive empty requests + empty flow = nil engine
-> `*TemplateExecuter.{Compile,Execute}` invoked
on nil engine = panic.
RCA:
1. `*ExecutorOptions.ApplyNewEngineOptions`
overwriting many fields.
2. copy op pointless; create a copy of options and
then immediately replace it with original
pointer.
3. missing executable requests validation after
cached templates is reconstructed with updated
options.
Thus, this affected `--automatic-scan` mode where
tech detection templates often have conditional
requests that may be filtered based on runtime
options.
Fixes#6417
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(templates): recompile workflow with `tplCopy.Options`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(templates): strengthen cache hit guard
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(protocols): skips template-specific fields
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
