The `race` condition directive was broken due to
a strict dependency on `threads > 0` for parallel
execution, causing templates with `race` directive
enabled but no explicit threads to fall back to
seq execution.
This regression was introduced in v3.2.0 (#4868),
which restricted parallel execution to only when
`payloads` were present.
Fixes#5713 to allow race conditions even w/o
explicit `payloads`, and add a default thread
count when race is enabled but threads is 0.
Signed-off-by: Dwi Siswanto <git@dw1.io>
Continue the fix from #6666 by converting
remaining direct Body assignments to use setter
methods:
* pkg/fuzz/component/body.go:139: use
`SetBodyReader()` in transfer-encoding path.
* pkg/protocols/http/request.go:694: use
`SetBodyString()` in fuzz component `Rebuild()`.
Fixes#6692.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(http): pass `dynamicValues` to `EvaluateWithInteractsh`
When `LazyEval` is true (triggered by `variables`
containing `BaseURL`, `Hostname`,
`interactsh-url`, etc.), variable expressions are not
eval'ed during YAML parsing & remain as raw exprs
like "{{rand_base(5)}}".
At request build time, `EvaluateWithInteractsh()`
checks if a variable already has a value in the
passed map before re-evaluating its expression.
But, `dynamicValues` (which contains the template
context with previously eval'ed values) was not
being passed, causing exprs like `rand_*` to be
re-evaluated on each request, producing different
values.
Fixes#6684 by including `dynamicValues` in the
map passed to `EvaluateWithInteractsh()`, so
variables evaluated in earlier requests retain
their values in subsequent requests.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(http): rm early eval in `(*Request).ExecuteWithResults()`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds variables-threads-previous integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds constants-with-threads integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds race-with-variables integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Prev, `FullResponseString()`, `BodyString()`, and
`HeadersString()` were called multiple times per
HTTP response iteration, each call allocating a
new string copy of the response data.
For a 10MB response, this resulted in ~60MB of
redundant string allocs/response (6 calls x 10MB).
Cache the string representations once per `Fill()`
cycle and reuse them throughout the response
processing loop. This reduces allocs from 6 to 3
per response, cutting memory usage by ~50% for
response string handling.
Profiling showed these functions accounting for
~89% of heap allocs (5.7GB out of 6.17GB) during
large scans.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(interactsh): skip DNS lookups on interactsh domains
to prevent false positives.
Prevents nuclei from resolving interactsh domains
injected in Host headers, which would cause
self-interactions to be incorrectly reported as
matches.
Changes:
* Add `GetHostname()` method to `interactsh.Client`
to expose active server domain.
* Skip CNAME DNS lookups in
`(*http.Request).addCNameIfAvailable` when
hostname matches the
`(*interactsh.Client).GetHostname`.
Fixes#6613
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(http): prevent false `interactshDomain` matches
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
across multiple layers
Fixes timeout configuration conflicts where HTTP
requests would timeout prematurely despite
configured values in `@timeout` annotations or
`-timeout` flags.
RCA:
* `retryablehttp` pkg overriding with default
30s timeout.
* Custom timeouts not propagating to
`retryablehttp` layer.
* Multiple timeout layers not sync properly.
Changes:
* Propagate custom timeouts from `@timeout`
annotations to `retryablehttp` layer.
* Adjust 5-minute maximum cap to prevent DoS via
extremely large timeouts.
* Ensure `retryableHttpOptions.Timeout` respects
`ResponseHeaderTimeout`.
* Add comprehensive tests for timeout capping
behavior.
This allows templates to override global timeout
via `@timeout` annotations while preventing abuse
thru unreasonably large timeout values.
Fixes#6560.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Enhance matcher compilation with caching for regex and DSL expressions to improve performance. Update template parsing to conditionally retain raw templates based on size constraints.
* Implement caching for regex and DSL expressions in extractors and matchers to enhance performance. Introduce a buffer pool in raw requests to reduce memory allocations. Update template cache management for improved efficiency.
* feat: improve concurrency to be bound
* refactor: replace fmt.Sprintf with fmt.Fprintf for improved performance in header handling
* feat: add regex matching tests and benchmarks for performance evaluation
* feat: add prefix check in regex extraction to optimize matching process
* feat: implement regex caching mechanism to enhance performance in extractors and matchers, along with tests and benchmarks for validation
* feat: add unit tests for template execution in the core engine, enhancing test coverage and reliability
* feat: enhance error handling in template execution and improve regex caching logic for better performance
* Implement caching for regex and DSL expressions in the cache package, replacing previous sync.Map usage. Add unit tests for cache functionality, including eviction by capacity and retrieval of cached items. Update extractors and matchers to utilize the new cache system for improved performance and memory efficiency.
* Add tests for SetCapacities in cache package to ensure cache behavior on capacity changes
- Implemented TestSetCapacities_NoRebuildOnZero to verify that setting capacities to zero does not clear existing caches.
- Added TestSetCapacities_BeforeFirstUse to confirm that initial cache settings are respected and not overridden by subsequent capacity changes.
* Refactor matchers and update load test generator to use io package
- Removed maxRegexScanBytes constant from match.go.
- Replaced ioutil with io package in load_test.go for NopCloser usage.
- Restored TestValidate_AllowsInlineMultiline in load_test.go to ensure inline validation functionality.
* Add cancellation support in template execution and enhance test coverage
- Updated executeTemplateWithTargets to respect context cancellation.
- Introduced fakeTargetProvider and slowExecuter for testing.
- Added Test_executeTemplateWithTargets_RespectsCancellation to validate cancellation behavior during template execution.
* Use proxy for dns and ssl templates
- while using template execute level function we need to override custom dialer
* rename overridedialer to customdialer
* Add proxy into hash
- proxy client is shared between non proxy requests
* add dialer into request object
- use request.dialer instead of global variable
* resolve comments
* rename dialer
* feat: fixed max-host-error blocking wrong port for template with error
* feat: log total results with time taken at end of execution
* bugfix: skip non-executed requests with progress in flow protocol
* feat: fixed request calculation in http protocol for progress
* misc adjustments
---------
Co-authored-by: Ice3man <nizamulrana@gmail.com>
* feat: added fuzzing output enhancements
* changes as requested
* misc
* feat: added dfp flag to display fuzz points + misc additions
* feat: added support for fuzzing nested path segments
* feat: added parts to fuzzing requests
* feat: added tracking for parameter occurence frequency in fuzzing
* added cli flag for fuzz frequency
* fixed broken tests
* fixed path based sqli integration test
* feat: added configurable fuzzing aggression level for payloads
* fixed failing test
* feat: added analyzers implementation for fuzzing
* feat: misc changes to analyzer
* feat: misc additions of units + tests fix
* misc changes to implementation
* feat: global matchers
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Ice3man543 <ice3man543@users.noreply.github.com>
* feat(globalmatchers): make `Callback` as type
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat: update `passive` term to `(matchers-)static`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): add `origin-template-*` event
also use `Set` method instead of `maps.Clone`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat: update `matchers-static` term to `global-matchers`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): clone event before `operator.Execute`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(tmplexec): don't store `matched` on `global-matchers` templ
This will end up generating 2 events from the same
`scan.ScanContext` if one of the templates has
`global-matchers` enabled. This way, non-
`global-matchers` templates can enter the
`writeFailureCallback` func to log failure output.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): initializes `requests` on `New`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): add `hasStorage` method
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(templates): rename global matchers checks method
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(loader): handle nil `templates.Template` pointer
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Ice3man543 <ice3man543@users.noreply.github.com>
added a check in `addCNameIfAvailable` to ensure
the `Dialer` isnot NIL before attempting to fetch
DNS data.
this prevents potential panics (ex. SIGSEGV) when
the `Dialer` is closed due to an interruption.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* more goroutine leak fixes to nuclei
* run only dns templates for test
* updated httpx to dev
* dep update
---------
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
* feat: added support for context cancellation to engine
* misc
* feat: added contexts everywhere
* misc
* misc
* use granular http timeouts and increase http timeout to 30s using multiplier
* track response header timeout in mhe
* update responseHeaderTimeout to 5sec
* skip failing windows test
---------
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
* validate and fix empty internal-event
* fix on error with interactsh req
* disable clustering in flow & multiproto
* fix empty/missing matcher-status result
* fix cluster unit test
* fix no results found unit test
