Compare commits

..

50 Commits

Author SHA1 Message Date
yyhuni
ab800eca06 feat(frontend): reorder navigation tabs for improved UX
- Move "Websites" tab to first position in scan history and target layouts
- Reposition "IP Addresses" tab before "Ports" for better logical flow
- Maintain consistent tab ordering across both scan history and target pages
- Improve navigation hierarchy by placing primary discovery results first
2026-01-01 09:47:30 +08:00
yyhuni
e8e5572339 perf(asset): add GIN indexes for tech array fields and improve query parser
- Add GinIndex for tech array field in Endpoint model to optimize __contains queries
- Add GinIndex for tech array field in WebSite model to optimize __contains queries
- Import GinIndex from django.contrib.postgres.indexes
- Refactor QueryParser to protect quoted filter values during tokenization
- Implement placeholder-based filter extraction to preserve spaces within quoted values
- Replace filter tokens with placeholders before logical operator normalization
- Restore original filter conditions from placeholders during parsing
- Fix spacing in comments for consistency (add space after "从")
- Improves query performance for technology stack filtering on large datasets
2026-01-01 08:58:03 +08:00
github-actions[bot]
d48d4bbcad chore: bump version to v1.2.12-dev 2025-12-31 16:01:48 +00:00
yyhuni
d1cca4c083 base timeout set 10s 2025-12-31 23:27:02 +08:00
yyhuni
df0810c863 feat: add fingerprint recognition feature and update documentation
- Add fingerprint recognition section to README with support for 2.7W+ rules from multiple sources (EHole, Goby, Wappalyzer, Fingers, FingerPrintHub, ARL)
- Update scanning pipeline architecture diagram to include fingerprint recognition stage between site identification and deep analysis
- Add fingerprint recognition styling to mermaid diagram for visual consistency
- Include WORKER_API_KEY environment variable in task distributor for worker authentication
- Update WeChat QR code image and public account name from "洋洋的小黑屋" to "塔罗安全学苑"
- Fix import statements in nav-system.tsx to use i18n navigation utilities instead of next/link and next/navigation
- Enhance scanning workflow documentation to reflect complete pipeline: subdomain discovery → port scanning → site identification → fingerprint recognition → URL collection → directory scanning → vulnerability scanning
2025-12-31 23:09:25 +08:00
yyhuni
d33e54c440 docs: simplify quick-start guide
- Remove alternative ZIP download method, keep only Git clone approach
- Remove update.sh script reference from service management section
- Remove dedicated "定期更新" (periodic updates) section
- Streamline documentation to focus on primary installation and usage paths
2025-12-31 22:50:08 +08:00
yyhuni
35a306fe8b fix:dev环境 2025-12-31 22:46:42 +08:00
yyhuni
724df82931 chore: pin Docker base image digests and add worker API key generation
- Pin golang:1.24 base image to specific digest to prevent upstream cache invalidation
- Pin ubuntu:24.04 base image to specific digest to prevent upstream cache invalidation
- Add WORKER_API_KEY generation in install.sh auto_fill_docker_env_secrets function
- Generate random 32-character string for WORKER_API_KEY during installation
- Update installation info message to include WORKER_API_KEY in generated secrets list
- Improve build reproducibility and security by using immutable image references
2025-12-31 22:40:38 +08:00
yyhuni
8dfffdf802 fix:认证 2025-12-31 22:21:40 +08:00
github-actions[bot]
b8cb85ce0b chore: bump version to v1.2.9-dev 2025-12-31 13:48:44 +00:00
yyhuni
da96d437a4 增加授权认证 2025-12-31 20:18:34 +08:00
github-actions[bot]
feaf8062e5 chore: bump version to v1.2.8-dev 2025-12-31 11:33:14 +00:00
yyhuni
4bab76f233 fix:组织删除问题 2025-12-31 17:50:37 +08:00
yyhuni
09416b4615 fix:redis端口 2025-12-31 17:45:25 +08:00
github-actions[bot]
bc1c5f6b0e chore: bump version to v1.2.7-dev 2025-12-31 06:16:42 +00:00
github-actions[bot]
2f2742e6fe chore: bump version to v1.2.6-dev 2025-12-31 05:29:36 +00:00
yyhuni
be3c346a74 增加搜索字段 2025-12-31 12:40:21 +08:00
yyhuni
0c7a6fff12 增加tech字段的搜索 2025-12-31 12:37:02 +08:00
yyhuni
3b4f0e3147 fix:指纹识别 2025-12-31 12:30:31 +08:00
yyhuni
51212a2a0c fix:指纹识别 2025-12-31 12:17:23 +08:00
yyhuni
58533bbaf6 fix:docker api 2025-12-31 12:03:08 +08:00
github-actions[bot]
6ccca1602d chore: bump version to v1.2.5-dev 2025-12-31 03:48:32 +00:00
yyhuni
6389b0f672 feat(fingerprints): Add type annotation to getAcceptConfig function
- Add explicit return type annotation `Record<string, string[]>` to getAcceptConfig function
- Improve type safety and IDE autocomplete for file type configuration
- Enhance code clarity for accepted file types mapping in import dialog
2025-12-31 10:17:25 +08:00
yyhuni
d7599b8599 feat(fingerprints): Add database indexes and expand test data generation
- Add database indexes on 'link' field in FingersFingerprint model for improved query performance
- Add database index on 'author' field in FingerPrintHubFingerprint model for filtering optimization
- Expand test data generation to include Fingers, FingerPrintHub, and ARL fingerprint types
- Add comprehensive fingerprint data generation methods with realistic templates and patterns
- Update test data cleanup to include all fingerprint table types
- Add i18n translations for fingerprint-related UI components and labels
- Optimize route prefetching hook for better performance
- Improve fingerprint data table columns and vulnerability columns display consistencyzxc
2025-12-31 10:04:15 +08:00
yyhuni
8eff298293 更新镜像加速逻辑 2025-12-31 08:56:55 +08:00
yyhuni
3634101c5b 添加灯塔等指纹 2025-12-31 08:55:37 +08:00
yyhuni
163973a7df feat(i18n): Add internationalization support to dropzone component
- Add useTranslations hook to DropzoneContent component for multi-language support
- Add useTranslations hook to DropzoneEmptyState component for multi-language support
- Replace hardcoded English strings with i18n translation keys in dropzone UI
- Add comprehensive translation keys for dropzone messages in en.json:
* uploadFile, uploadFiles, dragOrClick, dragOrClickReplace
* moreFiles, supports, minimum, maximum, sizeBetween
- Add corresponding Chinese translations in zh.json for all dropzone messages
- Support dynamic content in translations using parameterized keys (files count, size ranges)
- Ensure consistent user experience across English and Chinese interfaces
2025-12-30 21:19:37 +08:00
yyhuni
80ffecba3e feat(i18n): Add UI component i18n provider and standardize translation keys
- Add UiI18nProvider component to wrap UI library translations globally
- Integrate UiI18nProvider into root layout for consistent i18n support
- Standardize download action translation keys (allEndpoints → all, selectedEndpoints → selected)
- Update ExpandableTagList component prop from maxVisible to maxLines for better layout control
- Fix color scheme in dashboard stop scan button (chart-2 → primary)
- Add DOCKER_API_VERSION configuration to backend settings for Docker client compatibility
- Update task distributor to use configurable Docker API version (default 1.40)
- Add environment variable support for Docker API version in task execution commands
- Update i18n configuration and message files with standardized keys
- Ensure UI components respect application locale settings across all data tables and dialogs
2025-12-30 21:19:28 +08:00
yyhuni
3c21ac940c 恢复ssh docker 2025-12-30 20:35:51 +08:00
yyhuni
5c9f484d70 fix(frontend): Fix i18n translation key references and add missing labels
- Change "nav" translation namespace to "navigation" in scan engine and wordlists pages
- Replace parameterized translation calls with raw translation strings for cron schedule options in scheduled scan page and dashboard component
- Cast raw translation results to string type for proper TypeScript typing
- Add missing "name" and "type" labels to fingerprint section in English and Chinese message files
- Ensure consistent translation key usage across components for better maintainability
2025-12-30 18:21:16 +08:00
yyhuni
7567f6c25b 更新文字描述 2025-12-30 18:08:39 +08:00
yyhuni
0599a0b298 ansi-to-html加入 2025-12-30 18:01:29 +08:00
yyhuni
f7557fe90c ansi-to-html替代log显示 2025-12-30 18:01:22 +08:00
yyhuni
13571b9772 fix(frontend): Fix xterm SSR initialization error
- Add 100ms delay for terminal initialization to ensure DOM is mounted
- Use requestAnimationFrame for fit() to avoid dimensions error
- Add try-catch for all xterm operations
- Proper cleanup on unmount

Fixes: Cannot read properties of undefined (reading 'dimensions')
2025-12-30 17:41:38 +08:00
yyhuni
8ee76eef69 feat(frontend): Add ANSI color support for system logs
- Create AnsiLogViewer component using xterm.js
- Replace Monaco Editor with xterm for log viewing
- Native ANSI escape code rendering (colors, bold, etc.)
- Auto-scroll to bottom, clickable URLs support

Benefits:
- Colorized logs for better readability
- No more escape codes like [32m[0m in UI
- Professional terminal-like experience
2025-12-30 17:39:12 +08:00
yyhuni
2a31e29aa2 fix: Add shell quoting for command arguments
- Use shlex.quote() to escape special characters in argument values
- Fixes: 'unrecognized arguments' error when values contain spaces
- Example: target_name='example.com scan' now correctly quoted
2025-12-30 17:32:09 +08:00
yyhuni
81abc59961 Refactor: Migrate TaskDistributor to Docker SDK
- Replace CLI subprocess with Python Docker SDK
- Add DockerClientManager for unified container management
- Remove 300+ lines of shell command building code
- Enable future features: container status monitoring, log streaming

Breaking changes: None (backward compatible with existing scans)
Rollback: git reset --hard v1.0-before-docker-sdk
2025-12-30 17:23:18 +08:00
yyhuni
ffbfec6dd5 feat(stage2): Refactor TaskDistributor to use Docker SDK
- Replace CLI subprocess calls with DockerClientManager.run_container()
- Add helper methods: _build_container_command, _build_container_environment, _build_container_volumes
- Refactor execute_scan_flow() and execute_cleanup_on_all_workers() to use SDK
- Remove old CLI methods: _build_docker_command, _execute_docker_command, _execute_local_docker, _execute_ssh_docker
- Remove paramiko import (no longer needed for local workers)

Benefits:
- 300+ lines removed (CLI string building complexity)
- Type-safe container configuration (no more shlex.quote errors)
- Structured error handling (ImageNotFound, APIError)
- Ready for container status monitoring and log streaming
2025-12-30 17:20:26 +08:00
yyhuni
a0091636a8 feat(stage1): Add DockerClientManager
- Create docker_client_manager.py with local Docker client support
- Add container lifecycle management (run, status, logs, stop, remove)
- Implement structured error handling (ImageNotFound, APIError)
- Add client connection caching and reuse
- Set Docker API version to 1.40 (compatible with Docker 19.03+)
- Add dependencies: docker>=6.0.0, packaging>=21.0

TODO: Remote worker support (Docker Context or SSH tunnel)
2025-12-30 17:17:17 +08:00
yyhuni
69490ab396 feat: Add DockerClientManager for unified Docker client management
- Create docker_client_manager.py with local Docker client support
- Add container lifecycle management (run, status, logs, stop, remove)
- Implement structured error handling (ImageNotFound, APIError)
- Add client connection caching and reuse
- Set Docker API version to 1.40 (compatible with Docker 19.03+)
- Add docker>=6.0.0 and packaging>=21.0 dependencies

TODO: Remote worker support (Docker Context or SSH tunnel)
2025-12-30 17:15:29 +08:00
yyhuni
7306964abf 更新readme 2025-12-30 16:44:08 +08:00
yyhuni
cb6b0259e3 fix:响应不匹配 2025-12-30 16:40:17 +08:00
yyhuni
e1b4618e58 refactor(worker): isolate scan tools to dedicated directory
- Move scan tools base path from `/usr/local/bin` to `/opt/xingrin-tools/bin` to avoid conflicts with system tools and Python packages
- Create dedicated `/opt/xingrin-tools/bin` directory in worker Dockerfile following FHS standards
- Update PATH environment variable to prioritize project-specific tools directory
- Add `SCAN_TOOLS_PATH` environment variable to `.env.example` with documentation
- Update settings.py to use new default path with explanatory comments
- Fix TypeScript type annotation in system-logs-view.tsx for better maintainability
- Remove frontend package-lock.json to reduce repository size
- Update task distributor comment to reflect new tool location
This change improves tool isolation and prevents naming conflicts while maintaining FHS compliance.
2025-12-30 11:42:09 +08:00
yyhuni
556dcf5f62 重构日志ui功能 2025-12-30 11:13:38 +08:00
yyhuni
0628eef025 重构响应为标准响应格式 2025-12-30 10:56:26 +08:00
yyhuni
38ed8bc642 fix(scan): improve config parser validation and enable subdomain resolve timeout
- Uncomment timeout: auto setting in subdomain discovery config example
- Add validation to reject None or non-dict configuration values
- Raise ValueError with descriptive message when config is None
- Raise ValueError when config is not a dictionary type
- Update docstring to document Raises section for error conditions
- Prevent silent failures from malformed YAML configurations
2025-12-30 08:54:02 +08:00
yyhuni
2f4d6a2168 统一工具挂载为/usr/local/bin 2025-12-30 08:45:36 +08:00
yyhuni
c25cb9e06b fix:工具挂载 2025-12-30 08:39:17 +08:00
yyhuni
b14ab71c7f fix:auth frontend 2025-12-30 08:12:04 +08:00
github-actions[bot]
8b5060e2d3 chore: bump version to v1.2.2-dev 2025-12-29 17:08:05 +00:00
176 changed files with 28702 additions and 10987 deletions

View File

@@ -62,9 +62,14 @@
- **自定义流程** - YAML 配置扫描流程,灵活编排
- **定时扫描** - Cron 表达式配置,自动化周期扫描
### 🔖 指纹识别
- **多源指纹库** - 内置 EHole、Goby、Wappalyzer、Fingers、FingerPrintHub、ARL 等 2.7W+ 指纹规则
- **自动识别** - 扫描流程自动执行,识别 Web 应用技术栈
- **指纹管理** - 支持查询、导入、导出指纹规则
#### 扫描流程架构
完整的扫描流程包括子域名发现、端口扫描、站点发现、URL 收集、目录扫描、漏洞扫描等阶段
完整的扫描流程包括:子域名发现、端口扫描、站点发现、指纹识别、URL 收集、目录扫描、漏洞扫描等阶段
```mermaid
flowchart LR
@@ -75,7 +80,8 @@ flowchart LR
SUB["子域名发现<br/>subfinder, amass, puredns"]
PORT["端口扫描<br/>naabu"]
SITE["站点识别<br/>httpx"]
SUB --> PORT --> SITE
FINGER["指纹识别<br/>xingfinger"]
SUB --> PORT --> SITE --> FINGER
end
subgraph STAGE2["阶段 2: 深度分析"]
@@ -91,7 +97,7 @@ flowchart LR
FINISH["扫描完成"]
START --> STAGE1
SITE --> STAGE2
FINGER --> STAGE2
STAGE2 --> STAGE3
STAGE3 --> FINISH
@@ -103,6 +109,7 @@ flowchart LR
style SUB fill:#5dade2,stroke:#3498db,stroke-width:1px,color:#fff
style PORT fill:#5dade2,stroke:#3498db,stroke-width:1px,color:#fff
style SITE fill:#5dade2,stroke:#3498db,stroke-width:1px,color:#fff
style FINGER fill:#5dade2,stroke:#3498db,stroke-width:1px,color:#fff
style URL fill:#bb8fce,stroke:#9b59b6,stroke-width:1px,color:#fff
style DIR fill:#bb8fce,stroke:#9b59b6,stroke-width:1px,color:#fff
style VULN fill:#f0b27a,stroke:#e67e22,stroke-width:1px,color:#fff
@@ -178,7 +185,7 @@ cd xingrin
# 安装并启动(生产模式)
sudo ./install.sh
# 🇨🇳 中国大陆用户推荐使用镜像加速
# 🇨🇳 中国大陆用户推荐使用镜像加速(第三方加速服务可能会失效,不保证长期可用)
sudo ./install.sh --mirror
```
@@ -211,13 +218,12 @@ sudo ./uninstall.sh
- 🐛 **如果发现 Bug** 可以点击右边链接进行提交 [Issue](https://github.com/yyhuni/xingrin/issues)
- 💡 **有新想法比如UI设计功能设计等** 欢迎点击右边链接进行提交建议 [Issue](https://github.com/yyhuni/xingrin/issues)
- 🔧 **想参与开发?** 关注我公众号与我个人联系
## 📧 联系
- 目前版本就我个人使用,可能会有很多边界问题
- 如有问题,建议,其他,优先提交[Issue](https://github.com/yyhuni/xingrin/issues),也可以直接给我的公众号发消息,我都会回复的
- 微信公众号: **洋洋的小黑屋**
- 微信公众号: **塔罗安全学苑**
<img src="docs/wechat-qrcode.png" alt="微信公众号" width="200">

View File

@@ -1 +1 @@
v1.1.14
v1.2.12-dev

View File

@@ -1,6 +1,7 @@
from django.db import models
from django.contrib.postgres.fields import ArrayField
from django.contrib.postgres.indexes import GinIndex
from django.core.validators import MinValueValidator, MaxValueValidator
@@ -131,11 +132,12 @@ class Endpoint(models.Model):
ordering = ['-created_at']
indexes = [
models.Index(fields=['-created_at']),
models.Index(fields=['target']), # 优化从target_id快速查找下面的端点主关联字段
models.Index(fields=['target']), # 优化从 target_id快速查找下面的端点主关联字段
models.Index(fields=['url']), # URL索引优化查询性能
models.Index(fields=['host']), # host索引优化根据主机名查询
models.Index(fields=['status_code']), # 状态码索引,优化筛选
models.Index(fields=['title']), # title索引优化智能过滤搜索
GinIndex(fields=['tech']), # GIN索引优化 tech 数组字段的 __contains 查询
]
constraints = [
# 普通唯一约束url + target 组合唯一
@@ -229,9 +231,10 @@ class WebSite(models.Model):
models.Index(fields=['-created_at']),
models.Index(fields=['url']), # URL索引优化查询性能
models.Index(fields=['host']), # host索引优化根据主机名查询
models.Index(fields=['target']), # 优化从target_id快速查找下面的站点
models.Index(fields=['target']), # 优化从 target_id快速查找下面的站点
models.Index(fields=['title']), # title索引优化智能过滤搜索
models.Index(fields=['status_code']), # 状态码索引,优化智能过滤搜索
GinIndex(fields=['tech']), # GIN索引优化 tech 数组字段的 __contains 查询
]
constraints = [
# 普通唯一约束url + target 组合唯一

View File

@@ -28,6 +28,7 @@ class EndpointService:
'host': 'host',
'title': 'title',
'status': 'status_code',
'tech': 'tech',
}
def __init__(self):
@@ -115,7 +116,7 @@ class EndpointService:
"""获取目标下的所有端点"""
queryset = self.repo.get_by_target(target_id)
if filter_query:
queryset = apply_filters(queryset, filter_query, self.FILTER_FIELD_MAPPING)
queryset = apply_filters(queryset, filter_query, self.FILTER_FIELD_MAPPING, json_array_fields=['tech'])
return queryset
def count_endpoints_by_target(self, target_id: int) -> int:
@@ -134,7 +135,7 @@ class EndpointService:
"""获取所有端点(全局查询)"""
queryset = self.repo.get_all()
if filter_query:
queryset = apply_filters(queryset, filter_query, self.FILTER_FIELD_MAPPING)
queryset = apply_filters(queryset, filter_query, self.FILTER_FIELD_MAPPING, json_array_fields=['tech'])
return queryset
def iter_endpoint_urls_by_target(self, target_id: int, chunk_size: int = 1000) -> Iterator[str]:

View File

@@ -20,6 +20,7 @@ class WebSiteService:
'host': 'host',
'title': 'title',
'status': 'status_code',
'tech': 'tech',
}
def __init__(self, repository=None):
@@ -107,14 +108,14 @@ class WebSiteService:
"""获取目标下的所有网站"""
queryset = self.repo.get_by_target(target_id)
if filter_query:
queryset = apply_filters(queryset, filter_query, self.FILTER_FIELD_MAPPING)
queryset = apply_filters(queryset, filter_query, self.FILTER_FIELD_MAPPING, json_array_fields=['tech'])
return queryset
def get_all(self, filter_query: Optional[str] = None):
"""获取所有网站"""
queryset = self.repo.get_all()
if filter_query:
queryset = apply_filters(queryset, filter_query, self.FILTER_FIELD_MAPPING)
queryset = apply_filters(queryset, filter_query, self.FILTER_FIELD_MAPPING, json_array_fields=['tech'])
return queryset
def get_by_url(self, url: str, target_id: int) -> int:

View File

@@ -2,6 +2,8 @@ import logging
from rest_framework import viewsets, status, filters
from rest_framework.decorators import action
from rest_framework.response import Response
from apps.common.response_helpers import success_response, error_response
from apps.common.error_codes import ErrorCodes
from rest_framework.request import Request
from rest_framework.exceptions import NotFound, ValidationError as DRFValidationError
from django.core.exceptions import ValidationError, ObjectDoesNotExist
@@ -57,7 +59,7 @@ class AssetStatisticsViewSet(viewsets.ViewSet):
"""
try:
stats = self.service.get_statistics()
return Response({
return success_response(data={
'totalTargets': stats['total_targets'],
'totalSubdomains': stats['total_subdomains'],
'totalIps': stats['total_ips'],
@@ -80,9 +82,10 @@ class AssetStatisticsViewSet(viewsets.ViewSet):
})
except (DatabaseError, OperationalError) as e:
logger.exception("获取资产统计数据失败")
return Response(
{'error': '获取统计数据失败'},
status=status.HTTP_500_INTERNAL_SERVER_ERROR
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Failed to get statistics',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
@action(detail=False, methods=['get'], url_path='history')
@@ -107,12 +110,13 @@ class AssetStatisticsViewSet(viewsets.ViewSet):
days = min(max(days, 1), 90) # 限制在 1-90 天
history = self.service.get_statistics_history(days=days)
return Response(history)
return success_response(data=history)
except (DatabaseError, OperationalError) as e:
logger.exception("获取统计历史数据失败")
return Response(
{'error': '获取历史数据失败'},
status=status.HTTP_500_INTERNAL_SERVER_ERROR
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Failed to get history data',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
@@ -164,45 +168,50 @@ class SubdomainViewSet(viewsets.ModelViewSet):
响应:
{
"message": "批量创建完成",
"createdCount": 10,
"skippedCount": 2,
"invalidCount": 1,
"mismatchedCount": 1,
"totalReceived": 14
"data": {
"createdCount": 10,
"skippedCount": 2,
"invalidCount": 1,
"mismatchedCount": 1,
"totalReceived": 14
}
}
"""
from apps.targets.models import Target
target_pk = self.kwargs.get('target_pk')
if not target_pk:
return Response(
{'error': '必须在目标下批量创建子域名'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Must create subdomains under a target',
status_code=status.HTTP_400_BAD_REQUEST
)
# 获取目标
try:
target = Target.objects.get(pk=target_pk)
except Target.DoesNotExist:
return Response(
{'error': '目标不存在'},
status=status.HTTP_404_NOT_FOUND
return error_response(
code=ErrorCodes.NOT_FOUND,
message='Target not found',
status_code=status.HTTP_404_NOT_FOUND
)
# 验证目标类型必须为域名
if target.type != Target.TargetType.DOMAIN:
return Response(
{'error': '只有域名类型的目标支持导入子域名'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Only domain type targets support subdomain import',
status_code=status.HTTP_400_BAD_REQUEST
)
# 获取请求体中的子域名列表
subdomains = request.data.get('subdomains', [])
if not subdomains or not isinstance(subdomains, list):
return Response(
{'error': '请求体不能为空或格式错误'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Request body cannot be empty or invalid format',
status_code=status.HTTP_400_BAD_REQUEST
)
# 调用 service 层处理
@@ -214,19 +223,19 @@ class SubdomainViewSet(viewsets.ModelViewSet):
)
except Exception as e:
logger.exception("批量创建子域名失败")
return Response(
{'error': '服务器内部错误'},
status=status.HTTP_500_INTERNAL_SERVER_ERROR
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Server internal error',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
return Response({
'message': '批量创建完成',
return success_response(data={
'createdCount': result.created_count,
'skippedCount': result.skipped_count,
'invalidCount': result.invalid_count,
'mismatchedCount': result.mismatched_count,
'totalReceived': result.total_received,
}, status=status.HTTP_200_OK)
})
@action(detail=False, methods=['get'], url_path='export')
def export(self, request, **kwargs):
@@ -265,6 +274,7 @@ class WebSiteViewSet(viewsets.ModelViewSet):
- host="example" 主机名模糊匹配
- title="login" 标题模糊匹配
- status="200,301" 状态码多值匹配
- tech="nginx" 技术栈匹配(数组字段)
- 多条件空格分隔 AND 关系
"""
@@ -299,35 +309,38 @@ class WebSiteViewSet(viewsets.ModelViewSet):
响应:
{
"message": "批量创建完成",
"createdCount": 10,
"mismatchedCount": 2
"data": {
"createdCount": 10
}
}
"""
from apps.targets.models import Target
target_pk = self.kwargs.get('target_pk')
if not target_pk:
return Response(
{'error': '必须在目标下批量创建网站'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Must create websites under a target',
status_code=status.HTTP_400_BAD_REQUEST
)
# 获取目标
try:
target = Target.objects.get(pk=target_pk)
except Target.DoesNotExist:
return Response(
{'error': '目标不存在'},
status=status.HTTP_404_NOT_FOUND
return error_response(
code=ErrorCodes.NOT_FOUND,
message='Target not found',
status_code=status.HTTP_404_NOT_FOUND
)
# 获取请求体中的 URL 列表
urls = request.data.get('urls', [])
if not urls or not isinstance(urls, list):
return Response(
{'error': '请求体不能为空或格式错误'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Request body cannot be empty or invalid format',
status_code=status.HTTP_400_BAD_REQUEST
)
# 调用 service 层处理
@@ -340,15 +353,15 @@ class WebSiteViewSet(viewsets.ModelViewSet):
)
except Exception as e:
logger.exception("批量创建网站失败")
return Response(
{'error': '服务器内部错误'},
status=status.HTTP_500_INTERNAL_SERVER_ERROR
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Server internal error',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
return Response({
'message': '批量创建完成',
return success_response(data={
'createdCount': created_count,
}, status=status.HTTP_200_OK)
})
@action(detail=False, methods=['get'], url_path='export')
def export(self, request, **kwargs):
@@ -426,35 +439,38 @@ class DirectoryViewSet(viewsets.ModelViewSet):
响应:
{
"message": "批量创建完成",
"createdCount": 10,
"mismatchedCount": 2
"data": {
"createdCount": 10
}
}
"""
from apps.targets.models import Target
target_pk = self.kwargs.get('target_pk')
if not target_pk:
return Response(
{'error': '必须在目标下批量创建目录'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Must create directories under a target',
status_code=status.HTTP_400_BAD_REQUEST
)
# 获取目标
try:
target = Target.objects.get(pk=target_pk)
except Target.DoesNotExist:
return Response(
{'error': '目标不存在'},
status=status.HTTP_404_NOT_FOUND
return error_response(
code=ErrorCodes.NOT_FOUND,
message='Target not found',
status_code=status.HTTP_404_NOT_FOUND
)
# 获取请求体中的 URL 列表
urls = request.data.get('urls', [])
if not urls or not isinstance(urls, list):
return Response(
{'error': '请求体不能为空或格式错误'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Request body cannot be empty or invalid format',
status_code=status.HTTP_400_BAD_REQUEST
)
# 调用 service 层处理
@@ -467,15 +483,15 @@ class DirectoryViewSet(viewsets.ModelViewSet):
)
except Exception as e:
logger.exception("批量创建目录失败")
return Response(
{'error': '服务器内部错误'},
status=status.HTTP_500_INTERNAL_SERVER_ERROR
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Server internal error',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
return Response({
'message': '批量创建完成',
return success_response(data={
'createdCount': created_count,
}, status=status.HTTP_200_OK)
})
@action(detail=False, methods=['get'], url_path='export')
def export(self, request, **kwargs):
@@ -519,6 +535,7 @@ class EndpointViewSet(viewsets.ModelViewSet):
- host="example" 主机名模糊匹配
- title="login" 标题模糊匹配
- status="200,301" 状态码多值匹配
- tech="nginx" 技术栈匹配(数组字段)
- 多条件空格分隔 AND 关系
"""
@@ -553,35 +570,38 @@ class EndpointViewSet(viewsets.ModelViewSet):
响应:
{
"message": "批量创建完成",
"createdCount": 10,
"mismatchedCount": 2
"data": {
"createdCount": 10
}
}
"""
from apps.targets.models import Target
target_pk = self.kwargs.get('target_pk')
if not target_pk:
return Response(
{'error': '必须在目标下批量创建端点'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Must create endpoints under a target',
status_code=status.HTTP_400_BAD_REQUEST
)
# 获取目标
try:
target = Target.objects.get(pk=target_pk)
except Target.DoesNotExist:
return Response(
{'error': '目标不存在'},
status=status.HTTP_404_NOT_FOUND
return error_response(
code=ErrorCodes.NOT_FOUND,
message='Target not found',
status_code=status.HTTP_404_NOT_FOUND
)
# 获取请求体中的 URL 列表
urls = request.data.get('urls', [])
if not urls or not isinstance(urls, list):
return Response(
{'error': '请求体不能为空或格式错误'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Request body cannot be empty or invalid format',
status_code=status.HTTP_400_BAD_REQUEST
)
# 调用 service 层处理
@@ -594,15 +614,15 @@ class EndpointViewSet(viewsets.ModelViewSet):
)
except Exception as e:
logger.exception("批量创建端点失败")
return Response(
{'error': '服务器内部错误'},
status=status.HTTP_500_INTERNAL_SERVER_ERROR
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Server internal error',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
return Response({
'message': '批量创建完成',
return success_response(data={
'createdCount': created_count,
}, status=status.HTTP_200_OK)
})
@action(detail=False, methods=['get'], url_path='export')
def export(self, request, **kwargs):

View File

@@ -40,8 +40,14 @@ def fetch_config_and_setup_django():
print(f"[CONFIG] 正在从配置中心获取配置: {config_url}")
print(f"[CONFIG] IS_LOCAL={is_local}")
try:
# 构建请求头(包含 Worker API Key
headers = {}
worker_api_key = os.environ.get("WORKER_API_KEY", "")
if worker_api_key:
headers["X-Worker-API-Key"] = worker_api_key
# verify=False: 远程 Worker 通过 HTTPS 访问时可能使用自签名证书
resp = requests.get(config_url, timeout=10, verify=False)
resp = requests.get(config_url, headers=headers, timeout=10, verify=False)
resp.raise_for_status()
config = resp.json()
@@ -57,28 +63,17 @@ def fetch_config_and_setup_django():
os.environ.setdefault("DB_USER", db_user)
os.environ.setdefault("DB_PASSWORD", config['db']['password'])
# Redis 配置
os.environ.setdefault("REDIS_URL", config['redisUrl'])
# 日志配置
os.environ.setdefault("LOG_DIR", config['paths']['logs'])
os.environ.setdefault("LOG_LEVEL", config['logging']['level'])
os.environ.setdefault("ENABLE_COMMAND_LOGGING", str(config['logging']['enableCommandLogging']).lower())
os.environ.setdefault("DEBUG", str(config['debug']))
# Git 加速配置(用于 Git clone 加速)
git_mirror = config.get('gitMirror', '')
if git_mirror:
os.environ.setdefault("GIT_MIRROR", git_mirror)
print(f"[CONFIG] ✓ 配置获取成功")
print(f"[CONFIG] DB_HOST: {db_host}")
print(f"[CONFIG] DB_PORT: {db_port}")
print(f"[CONFIG] DB_NAME: {db_name}")
print(f"[CONFIG] DB_USER: {db_user}")
print(f"[CONFIG] REDIS_URL: {config['redisUrl']}")
if git_mirror:
print(f"[CONFIG] GIT_MIRROR: {git_mirror}")
except Exception as e:
print(f"[ERROR] 获取配置失败: {config_url} - {e}", file=sys.stderr)

View File

@@ -0,0 +1,31 @@
"""
标准化错误码定义
采用简化方案(参考 Stripe、GitHub 等大厂做法):
- 只定义 5-10 个通用错误码
- 未知错误使用通用错误码
- 错误码格式:大写字母和下划线组成
"""
class ErrorCodes:
"""标准化错误码
只定义通用错误码,其他错误使用通用消息。
这是 Stripe、GitHub 等大厂的标准做法。
错误码格式规范:
- 使用大写字母和下划线
- 简洁明了,易于理解
- 前端通过错误码映射到 i18n 键
"""
# 通用错误码8 个)
VALIDATION_ERROR = 'VALIDATION_ERROR' # 输入验证失败
NOT_FOUND = 'NOT_FOUND' # 资源未找到
PERMISSION_DENIED = 'PERMISSION_DENIED' # 权限不足
SERVER_ERROR = 'SERVER_ERROR' # 服务器内部错误
BAD_REQUEST = 'BAD_REQUEST' # 请求格式错误
CONFLICT = 'CONFLICT' # 资源冲突(如重复创建)
UNAUTHORIZED = 'UNAUTHORIZED' # 未认证
RATE_LIMITED = 'RATE_LIMITED' # 请求过于频繁

View File

@@ -0,0 +1,49 @@
"""
自定义异常处理器
统一处理 DRF 异常,确保错误响应格式一致
"""
from rest_framework.views import exception_handler
from rest_framework import status
from rest_framework.exceptions import AuthenticationFailed, NotAuthenticated
from apps.common.response_helpers import error_response
from apps.common.error_codes import ErrorCodes
def custom_exception_handler(exc, context):
"""
自定义异常处理器
处理认证相关异常,返回统一格式的错误响应
"""
# 先调用 DRF 默认的异常处理器
response = exception_handler(exc, context)
if response is not None:
# 处理 401 未认证错误
if response.status_code == status.HTTP_401_UNAUTHORIZED:
return error_response(
code=ErrorCodes.UNAUTHORIZED,
message='Authentication required',
status_code=status.HTTP_401_UNAUTHORIZED
)
# 处理 403 权限不足错误
if response.status_code == status.HTTP_403_FORBIDDEN:
return error_response(
code=ErrorCodes.PERMISSION_DENIED,
message='Permission denied',
status_code=status.HTTP_403_FORBIDDEN
)
# 处理 NotAuthenticated 和 AuthenticationFailed 异常
if isinstance(exc, (NotAuthenticated, AuthenticationFailed)):
return error_response(
code=ErrorCodes.UNAUTHORIZED,
message='Authentication required',
status_code=status.HTTP_401_UNAUTHORIZED
)
return response

View File

@@ -0,0 +1,80 @@
"""
集中式权限管理
实现三类端点的认证逻辑:
1. 公开端点(无需认证):登录、登出、获取当前用户状态
2. Worker 端点API Key 认证):注册、配置、心跳、回调、资源同步
3. 业务端点Session 认证):其他所有 API
"""
import re
import logging
from django.conf import settings
from rest_framework.permissions import BasePermission
logger = logging.getLogger(__name__)
# 公开端点白名单(无需任何认证)
PUBLIC_ENDPOINTS = [
r'^/api/auth/login/$',
r'^/api/auth/logout/$',
r'^/api/auth/me/$',
]
# Worker API 端点(需要 API Key 认证)
# 包括:注册、配置、心跳、回调、资源同步(字典下载)
WORKER_ENDPOINTS = [
r'^/api/workers/register/$',
r'^/api/workers/config/$',
r'^/api/workers/\d+/heartbeat/$',
r'^/api/callbacks/',
# 资源同步端点Worker 需要下载字典文件)
r'^/api/wordlists/download/$',
# 注意:指纹导出 API 使用 Session 认证(前端用户导出用)
# Worker 通过数据库直接获取指纹数据,不需要 HTTP API
]
class IsAuthenticatedOrPublic(BasePermission):
"""
自定义权限类:
- 白名单内的端点公开访问
- Worker 端点需要 API Key 认证
- 其他端点需要 Session 认证
"""
def has_permission(self, request, view):
path = request.path
# 检查是否在公开白名单内
for pattern in PUBLIC_ENDPOINTS:
if re.match(pattern, path):
return True
# 检查是否是 Worker 端点
for pattern in WORKER_ENDPOINTS:
if re.match(pattern, path):
return self._check_worker_api_key(request)
# 其他路径需要 Session 认证
return request.user and request.user.is_authenticated
def _check_worker_api_key(self, request):
"""验证 Worker API Key"""
api_key = request.headers.get('X-Worker-API-Key')
expected_key = getattr(settings, 'WORKER_API_KEY', None)
if not expected_key:
# 未配置 API Key 时,拒绝所有 Worker 请求
logger.warning("WORKER_API_KEY 未配置,拒绝 Worker 请求")
return False
if not api_key:
logger.warning(f"Worker 请求缺少 X-Worker-API-Key Header: {request.path}")
return False
if api_key != expected_key:
logger.warning(f"Worker API Key 无效: {request.path}")
return False
return True

View File

@@ -0,0 +1,88 @@
"""
标准化 API 响应辅助函数
遵循行业标准RFC 9457 Problem Details和大厂实践Google、Stripe、GitHub
- 成功响应只包含数据,不包含 message 字段
- 错误响应使用机器可读的错误码,前端映射到 i18n 消息
"""
from typing import Any, Dict, List, Optional, Union
from rest_framework import status
from rest_framework.response import Response
def success_response(
data: Optional[Union[Dict[str, Any], List[Any]]] = None,
status_code: int = status.HTTP_200_OK
) -> Response:
"""
标准化成功响应
直接返回数据,不做包装,符合 Stripe/GitHub 等大厂标准。
Args:
data: 响应数据dict 或 list
status_code: HTTP 状态码,默认 200
Returns:
Response: DRF Response 对象
Examples:
# 单个资源
>>> success_response(data={'id': 1, 'name': 'Test'})
{'id': 1, 'name': 'Test'}
# 操作结果
>>> success_response(data={'count': 3, 'scans': [...]})
{'count': 3, 'scans': [...]}
# 创建资源
>>> success_response(data={'id': 1}, status_code=201)
"""
# 注意:不能使用 data or {},因为空列表 [] 会被转换为 {}
if data is None:
data = {}
return Response(data, status=status_code)
def error_response(
code: str,
message: Optional[str] = None,
details: Optional[List[Dict[str, Any]]] = None,
status_code: int = status.HTTP_400_BAD_REQUEST
) -> Response:
"""
标准化错误响应
Args:
code: 错误码(如 'VALIDATION_ERROR', 'NOT_FOUND'
格式:大写字母和下划线组成
message: 开发者调试信息(非用户显示)
details: 详细错误信息(如字段级验证错误)
status_code: HTTP 状态码,默认 400
Returns:
Response: DRF Response 对象
Examples:
# 简单错误
>>> error_response(code='NOT_FOUND', status_code=404)
{'error': {'code': 'NOT_FOUND'}}
# 带调试信息
>>> error_response(
... code='VALIDATION_ERROR',
... message='Invalid input data',
... details=[{'field': 'name', 'message': 'Required'}]
... )
{'error': {'code': 'VALIDATION_ERROR', 'message': '...', 'details': [...]}}
"""
error_body: Dict[str, Any] = {'code': code}
if message:
error_body['message'] = message
if details:
error_body['details'] = details
return Response({'error': error_body}, status=status_code)

View File

@@ -4,15 +4,28 @@
提供系统日志的读取功能,支持:
- 从日志目录读取日志文件
- 限制返回行数,防止内存溢出
- 列出可用的日志文件
"""
import fnmatch
import logging
import os
import subprocess
from datetime import datetime, timezone
from typing import TypedDict
logger = logging.getLogger(__name__)
class LogFileInfo(TypedDict):
"""日志文件信息"""
filename: str
category: str # 'system' | 'error' | 'performance' | 'container'
size: int
modifiedAt: str # ISO 8601 格式
class SystemLogService:
"""
系统日志服务类
@@ -20,23 +33,131 @@ class SystemLogService:
负责读取系统日志文件,支持从容器内路径或宿主机挂载路径读取日志。
"""
# 日志文件分类规则
CATEGORY_RULES = [
('xingrin.log', 'system'),
('xingrin_error.log', 'error'),
('performance.log', 'performance'),
('container_*.log', 'container'),
]
def __init__(self):
# 日志文件路径(统一使用 /opt/xingrin/logs
self.log_file = "/opt/xingrin/logs/xingrin.log"
self.default_lines = 200 # 默认返回行数
self.max_lines = 10000 # 最大返回行数限制
self.timeout_seconds = 3 # tail 命令超时时间
# 日志目录路径
self.log_dir = "/opt/xingrin/logs"
self.default_file = "xingrin.log" # 默认日志文件
self.default_lines = 200 # 默认返回行数
self.max_lines = 10000 # 最大返回行数限制
self.timeout_seconds = 3 # tail 命令超时时间
def get_logs_content(self, lines: int | None = None) -> str:
def _categorize_file(self, filename: str) -> str | None:
"""
根据文件名判断日志分类
Returns:
分类名称,如果不是日志文件则返回 None
"""
for pattern, category in self.CATEGORY_RULES:
if fnmatch.fnmatch(filename, pattern):
return category
return None
def _validate_filename(self, filename: str) -> bool:
"""
验证文件名是否合法(防止路径遍历攻击)
Args:
filename: 要验证的文件名
Returns:
bool: 文件名是否合法
"""
# 不允许包含路径分隔符
if '/' in filename or '\\' in filename:
return False
# 不允许 .. 路径遍历
if '..' in filename:
return False
# 必须是已知的日志文件类型
return self._categorize_file(filename) is not None
def get_log_files(self) -> list[LogFileInfo]:
"""
获取所有可用的日志文件列表
Returns:
日志文件信息列表,按分类和文件名排序
"""
files: list[LogFileInfo] = []
if not os.path.isdir(self.log_dir):
logger.warning("日志目录不存在: %s", self.log_dir)
return files
for filename in os.listdir(self.log_dir):
filepath = os.path.join(self.log_dir, filename)
# 只处理文件,跳过目录
if not os.path.isfile(filepath):
continue
# 判断分类
category = self._categorize_file(filename)
if category is None:
continue
# 获取文件信息
try:
stat = os.stat(filepath)
modified_at = datetime.fromtimestamp(
stat.st_mtime, tz=timezone.utc
).isoformat()
files.append({
'filename': filename,
'category': category,
'size': stat.st_size,
'modifiedAt': modified_at,
})
except OSError as e:
logger.warning("获取文件信息失败 %s: %s", filepath, e)
continue
# 排序按分类优先级system > error > performance > container然后按文件名
category_order = {'system': 0, 'error': 1, 'performance': 2, 'container': 3}
files.sort(key=lambda f: (category_order.get(f['category'], 99), f['filename']))
return files
def get_logs_content(self, filename: str | None = None, lines: int | None = None) -> str:
"""
获取系统日志内容
Args:
filename: 日志文件名,默认为 xingrin.log
lines: 返回的日志行数,默认 200 行,最大 10000 行
Returns:
str: 日志内容,每行以换行符分隔,保持原始顺序
Raises:
ValueError: 文件名不合法
FileNotFoundError: 日志文件不存在
"""
# 文件名处理
if filename is None:
filename = self.default_file
# 验证文件名
if not self._validate_filename(filename):
raise ValueError(f"无效的文件名: {filename}")
# 构建完整路径
log_file = os.path.join(self.log_dir, filename)
# 检查文件是否存在
if not os.path.isfile(log_file):
raise FileNotFoundError(f"日志文件不存在: {filename}")
# 参数校验和默认值处理
if lines is None:
lines = self.default_lines
@@ -48,7 +169,7 @@ class SystemLogService:
lines = self.max_lines
# 使用 tail 命令读取日志文件末尾内容
cmd = ["tail", "-n", str(lines), self.log_file]
cmd = ["tail", "-n", str(lines), log_file]
result = subprocess.run(
cmd,

View File

@@ -2,14 +2,18 @@
通用模块 URL 配置
路由说明:
- /api/health/ 健康检查接口(无需认证)
- /api/auth/* 认证相关接口(登录、登出、用户信息)
- /api/system/* 系统管理接口(日志查看等)
"""
from django.urls import path
from .views import LoginView, LogoutView, MeView, ChangePasswordView, SystemLogsView
from .views import LoginView, LogoutView, MeView, ChangePasswordView, SystemLogsView, SystemLogFilesView, HealthCheckView
urlpatterns = [
# 健康检查(无需认证)
path('health/', HealthCheckView.as_view(), name='health-check'),
# 认证相关
path('auth/login/', LoginView.as_view(), name='auth-login'),
path('auth/logout/', LogoutView.as_view(), name='auth-logout'),
@@ -18,4 +22,5 @@ urlpatterns = [
# 系统管理
path('system/logs/', SystemLogsView.as_view(), name='system-logs'),
path('system/logs/files/', SystemLogFilesView.as_view(), name='system-log-files'),
]

View File

@@ -13,7 +13,6 @@ from .csv_utils import (
format_datetime,
UTF8_BOM,
)
from .git_proxy import get_git_proxy_url
__all__ = [
'deduplicate_for_bulk',
@@ -26,5 +25,4 @@ __all__ = [
'format_list_field',
'format_datetime',
'UTF8_BOM',
'get_git_proxy_url',
]

View File

@@ -86,9 +86,21 @@ class QueryParser:
if not query_string or not query_string.strip():
return []
# 第一步:提取所有过滤条件并用占位符替换,保护引号内的空格
filters_found = []
placeholder_pattern = '__FILTER_{}__'
def replace_filter(match):
idx = len(filters_found)
filters_found.append(match.group(0))
return placeholder_pattern.format(idx)
# 先用正则提取所有 field="value" 形式的条件
protected = cls.FILTER_PATTERN.sub(replace_filter, query_string)
# 标准化逻辑运算符
# 先处理 || 和 or -> __OR__
normalized = cls.OR_PATTERN.sub(' __OR__ ', query_string)
normalized = cls.OR_PATTERN.sub(' __OR__ ', protected)
# 再处理 && 和 and -> __AND__
normalized = cls.AND_PATTERN.sub(' __AND__ ', normalized)
@@ -103,20 +115,26 @@ class QueryParser:
pending_op = LogicalOp.OR
elif token == '__AND__':
pending_op = LogicalOp.AND
else:
# 尝试解析为过滤条件
match = cls.FILTER_PATTERN.match(token)
if match:
field, operator, value = match.groups()
groups.append(FilterGroup(
filter=ParsedFilter(
field=field.lower(),
operator=operator,
value=value
),
logical_op=pending_op if groups else LogicalOp.AND # 第一个条件默认 AND
))
pending_op = LogicalOp.AND # 重置为默认 AND
elif token.startswith('__FILTER_') and token.endswith('__'):
# 还原占位符为原始过滤条件
try:
idx = int(token[9:-2]) # 提取索引
original_filter = filters_found[idx]
match = cls.FILTER_PATTERN.match(original_filter)
if match:
field, operator, value = match.groups()
groups.append(FilterGroup(
filter=ParsedFilter(
field=field.lower(),
operator=operator,
value=value
),
logical_op=pending_op if groups else LogicalOp.AND
))
pending_op = LogicalOp.AND # 重置为默认 AND
except (ValueError, IndexError):
pass
# 其他 token 忽略(无效输入)
return groups

View File

@@ -1,39 +0,0 @@
"""Git proxy utilities for URL acceleration."""
import os
from urllib.parse import urlparse
def get_git_proxy_url(original_url: str) -> str:
"""
Convert Git repository URL to proxy format for acceleration.
Supports multiple mirror services (standard format):
- gh-proxy.org: https://gh-proxy.org/https://github.com/user/repo.git
- ghproxy.com: https://ghproxy.com/https://github.com/user/repo.git
- mirror.ghproxy.com: https://mirror.ghproxy.com/https://github.com/user/repo.git
- ghps.cc: https://ghps.cc/https://github.com/user/repo.git
Args:
original_url: Original repository URL, e.g., https://github.com/user/repo.git
Returns:
Converted URL based on GIT_MIRROR setting.
If GIT_MIRROR is not set, returns the original URL unchanged.
"""
git_mirror = os.getenv("GIT_MIRROR", "").strip()
if not git_mirror:
return original_url
# Remove trailing slash from mirror URL if present
git_mirror = git_mirror.rstrip("/")
parsed = urlparse(original_url)
host = parsed.netloc.lower()
# Only support GitHub for now
if "github.com" not in host:
return original_url
# Standard format: https://mirror.example.com/https://github.com/user/repo.git
return f"{git_mirror}/{original_url}"

View File

@@ -2,11 +2,17 @@
通用模块视图导出
包含:
- 健康检查视图Docker 健康检查
- 认证相关视图:登录、登出、用户信息、修改密码
- 系统日志视图:实时日志查看
"""
from .health_views import HealthCheckView
from .auth_views import LoginView, LogoutView, MeView, ChangePasswordView
from .system_log_views import SystemLogsView
from .system_log_views import SystemLogsView, SystemLogFilesView
__all__ = ['LoginView', 'LogoutView', 'MeView', 'ChangePasswordView', 'SystemLogsView']
__all__ = [
'HealthCheckView',
'LoginView', 'LogoutView', 'MeView', 'ChangePasswordView',
'SystemLogsView', 'SystemLogFilesView',
]

View File

@@ -9,7 +9,10 @@ from django.utils.decorators import method_decorator
from rest_framework import status
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.permissions import AllowAny, IsAuthenticated
from rest_framework.permissions import AllowAny
from apps.common.response_helpers import success_response, error_response
from apps.common.error_codes import ErrorCodes
logger = logging.getLogger(__name__)
@@ -28,9 +31,10 @@ class LoginView(APIView):
password = request.data.get('password')
if not username or not password:
return Response(
{'error': '请提供用户名和密码'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Username and password are required',
status_code=status.HTTP_400_BAD_REQUEST
)
user = authenticate(request, username=username, password=password)
@@ -38,20 +42,22 @@ class LoginView(APIView):
if user is not None:
login(request, user)
logger.info(f"用户 {username} 登录成功")
return Response({
'message': '登录成功',
'user': {
'id': user.id,
'username': user.username,
'isStaff': user.is_staff,
'isSuperuser': user.is_superuser,
return success_response(
data={
'user': {
'id': user.id,
'username': user.username,
'isStaff': user.is_staff,
'isSuperuser': user.is_superuser,
}
}
})
)
else:
logger.warning(f"用户 {username} 登录失败:用户名或密码错误")
return Response(
{'error': '用户名或密码错误'},
status=status.HTTP_401_UNAUTHORIZED
return error_response(
code=ErrorCodes.UNAUTHORIZED,
message='Invalid username or password',
status_code=status.HTTP_401_UNAUTHORIZED
)
@@ -79,7 +85,7 @@ class LogoutView(APIView):
logout(request)
else:
logout(request)
return Response({'message': '已登出'})
return success_response()
@method_decorator(csrf_exempt, name='dispatch')
@@ -100,22 +106,26 @@ class MeView(APIView):
if user_id:
try:
user = User.objects.get(pk=user_id)
return Response({
'authenticated': True,
'user': {
'id': user.id,
'username': user.username,
'isStaff': user.is_staff,
'isSuperuser': user.is_superuser,
return success_response(
data={
'authenticated': True,
'user': {
'id': user.id,
'username': user.username,
'isStaff': user.is_staff,
'isSuperuser': user.is_superuser,
}
}
})
)
except User.DoesNotExist:
pass
return Response({
'authenticated': False,
'user': None
})
return success_response(
data={
'authenticated': False,
'user': None
}
)
@method_decorator(csrf_exempt, name='dispatch')
@@ -124,43 +134,27 @@ class ChangePasswordView(APIView):
修改密码
POST /api/auth/change-password/
"""
authentication_classes = [] # 禁用认证(绕过 CSRF
permission_classes = [AllowAny] # 手动检查登录状态
def post(self, request):
# 手动检查登录状态(从 session 获取用户
from django.contrib.auth import get_user_model
User = get_user_model()
user_id = request.session.get('_auth_user_id')
if not user_id:
return Response(
{'error': '请先登录'},
status=status.HTTP_401_UNAUTHORIZED
)
try:
user = User.objects.get(pk=user_id)
except User.DoesNotExist:
return Response(
{'error': '用户不存在'},
status=status.HTTP_401_UNAUTHORIZED
)
# 使用全局权限类验证request.user 已经是认证用户
user = request.user
# CamelCaseParser 将 oldPassword -> old_password
old_password = request.data.get('old_password')
new_password = request.data.get('new_password')
if not old_password or not new_password:
return Response(
{'error': '请提供旧密码和新密码'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Old password and new password are required',
status_code=status.HTTP_400_BAD_REQUEST
)
if not user.check_password(old_password):
return Response(
{'error': '旧密码错误'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Old password is incorrect',
status_code=status.HTTP_400_BAD_REQUEST
)
user.set_password(new_password)
@@ -170,4 +164,4 @@ class ChangePasswordView(APIView):
update_session_auth_hash(request, user)
logger.info(f"用户 {user.username} 已修改密码")
return Response({'message': '密码修改成功'})
return success_response()

View File

@@ -0,0 +1,24 @@
"""
健康检查视图
提供 Docker 健康检查端点,无需认证。
"""
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.permissions import AllowAny
class HealthCheckView(APIView):
"""
健康检查端点
GET /api/health/
返回服务状态,用于 Docker 健康检查。
此端点无需认证。
"""
permission_classes = [AllowAny]
def get(self, request):
return Response({'status': 'ok'})

View File

@@ -9,16 +9,57 @@ import logging
from django.utils.decorators import method_decorator
from django.views.decorators.csrf import csrf_exempt
from rest_framework import status
from rest_framework.permissions import AllowAny
from rest_framework.response import Response
from rest_framework.views import APIView
from apps.common.response_helpers import success_response, error_response
from apps.common.error_codes import ErrorCodes
from apps.common.services.system_log_service import SystemLogService
logger = logging.getLogger(__name__)
@method_decorator(csrf_exempt, name="dispatch")
class SystemLogFilesView(APIView):
"""
日志文件列表 API 视图
GET /api/system/logs/files/
获取所有可用的日志文件列表
Response:
{
"files": [
{
"filename": "xingrin.log",
"category": "system",
"size": 1048576,
"modifiedAt": "2025-01-15T10:30:00+00:00"
},
...
]
}
"""
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.service = SystemLogService()
def get(self, request):
"""获取日志文件列表"""
try:
files = self.service.get_log_files()
return success_response(data={"files": files})
except Exception:
logger.exception("获取日志文件列表失败")
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Failed to get log files',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
@method_decorator(csrf_exempt, name="dispatch")
class SystemLogsView(APIView):
"""
@@ -28,21 +69,14 @@ class SystemLogsView(APIView):
获取系统日志内容
Query Parameters:
file (str, optional): 日志文件名,默认 xingrin.log
lines (int, optional): 返回的日志行数,默认 200最大 10000
Response:
{
"content": "日志内容字符串..."
}
Note:
- 当前为开发阶段,暂时允许匿名访问
- 生产环境应添加管理员权限验证
"""
# TODO: 生产环境应改为 IsAdminUser 权限
authentication_classes = []
permission_classes = [AllowAny]
def __init__(self, **kwargs):
super().__init__(**kwargs)
@@ -52,18 +86,33 @@ class SystemLogsView(APIView):
"""
获取系统日志
支持通过 lines 参数控制返回行数,用于前端分页或实时刷新场景
支持通过 file 和 lines 参数控制返回内容
"""
try:
# 解析 lines 参数
# 解析参数
filename = request.query_params.get("file")
lines_raw = request.query_params.get("lines")
lines = int(lines_raw) if lines_raw is not None else None
# 调用服务获取日志内容
content = self.service.get_logs_content(lines=lines)
return Response({"content": content})
except ValueError:
return Response({"error": "lines 参数必须是整数"}, status=status.HTTP_400_BAD_REQUEST)
content = self.service.get_logs_content(filename=filename, lines=lines)
return success_response(data={"content": content})
except ValueError as e:
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message=str(e) if 'file' in str(e).lower() else 'lines must be an integer',
status_code=status.HTTP_400_BAD_REQUEST
)
except FileNotFoundError as e:
return error_response(
code=ErrorCodes.NOT_FOUND,
message=str(e),
status_code=status.HTTP_404_NOT_FOUND
)
except Exception:
logger.exception("获取系统日志失败")
return Response({"error": "获取系统日志失败"}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Failed to get system logs',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)

View File

@@ -0,0 +1,44 @@
"""
WebSocket 认证基类
提供需要认证的 WebSocket Consumer 基类
"""
import logging
from channels.generic.websocket import AsyncWebsocketConsumer
logger = logging.getLogger(__name__)
class AuthenticatedWebsocketConsumer(AsyncWebsocketConsumer):
"""
需要认证的 WebSocket Consumer 基类
子类应该重写 on_connect() 方法实现具体的连接逻辑
"""
async def connect(self):
"""
连接时验证用户认证状态
未认证时使用 close(code=4001) 拒绝连接
"""
user = self.scope.get('user')
if not user or not user.is_authenticated:
logger.warning(
f"WebSocket 连接被拒绝:用户未认证 - Path: {self.scope.get('path')}"
)
await self.close(code=4001)
return
# 调用子类的连接逻辑
await self.on_connect()
async def on_connect(self):
"""
子类实现具体的连接逻辑
默认实现:接受连接
"""
await self.accept()

View File

@@ -6,17 +6,17 @@ import json
import logging
import asyncio
import os
from channels.generic.websocket import AsyncWebsocketConsumer
from asgiref.sync import sync_to_async
from django.conf import settings
from apps.common.websocket_auth import AuthenticatedWebsocketConsumer
from apps.engine.services import WorkerService
logger = logging.getLogger(__name__)
class WorkerDeployConsumer(AsyncWebsocketConsumer):
class WorkerDeployConsumer(AuthenticatedWebsocketConsumer):
"""
Worker 交互式终端 WebSocket Consumer
@@ -31,8 +31,8 @@ class WorkerDeployConsumer(AsyncWebsocketConsumer):
self.read_task = None
self.worker_service = WorkerService()
async def connect(self):
"""连接时加入对应 Worker 的组并自动建立 SSH 连接"""
async def on_connect(self):
"""连接时加入对应 Worker 的组并自动建立 SSH 连接(已通过认证)"""
self.worker_id = self.scope['url_route']['kwargs']['worker_id']
self.group_name = f'worker_deploy_{self.worker_id}'

View File

@@ -90,6 +90,7 @@ class Command(BaseCommand):
single_config,
sort_keys=False,
allow_unicode=True,
default_flow_style=None,
)
except yaml.YAMLError as e:
self.stdout.write(self.style.ERROR(f'生成子引擎 {scan_type} 配置失败: {e}'))

View File

@@ -3,6 +3,9 @@
- EHole 指纹: ehole.json -> 导入到数据库
- Goby 指纹: goby.json -> 导入到数据库
- Wappalyzer 指纹: wappalyzer.json -> 导入到数据库
- Fingers 指纹: fingers_http.json -> 导入到数据库
- FingerPrintHub 指纹: fingerprinthub_web.json -> 导入到数据库
- ARL 指纹: ARL.yaml -> 导入到数据库
可重复执行:如果数据库已有数据则跳过,只在空库时导入。
"""
@@ -11,14 +14,25 @@ import json
import logging
from pathlib import Path
import yaml
from django.conf import settings
from django.core.management.base import BaseCommand
from apps.engine.models import EholeFingerprint, GobyFingerprint, WappalyzerFingerprint
from apps.engine.models import (
EholeFingerprint,
GobyFingerprint,
WappalyzerFingerprint,
FingersFingerprint,
FingerPrintHubFingerprint,
ARLFingerprint,
)
from apps.engine.services.fingerprints import (
EholeFingerprintService,
GobyFingerprintService,
WappalyzerFingerprintService,
FingersFingerprintService,
FingerPrintHubFingerprintService,
ARLFingerprintService,
)
@@ -33,6 +47,7 @@ DEFAULT_FINGERPRINTS = [
"model": EholeFingerprint,
"service": EholeFingerprintService,
"data_key": "fingerprint", # JSON 中指纹数组的 key
"file_format": "json",
},
{
"type": "goby",
@@ -40,6 +55,7 @@ DEFAULT_FINGERPRINTS = [
"model": GobyFingerprint,
"service": GobyFingerprintService,
"data_key": None, # Goby 是数组格式,直接使用整个 JSON
"file_format": "json",
},
{
"type": "wappalyzer",
@@ -47,6 +63,31 @@ DEFAULT_FINGERPRINTS = [
"model": WappalyzerFingerprint,
"service": WappalyzerFingerprintService,
"data_key": "apps", # Wappalyzer 使用 apps 对象
"file_format": "json",
},
{
"type": "fingers",
"filename": "fingers_http.json",
"model": FingersFingerprint,
"service": FingersFingerprintService,
"data_key": None, # Fingers 是数组格式
"file_format": "json",
},
{
"type": "fingerprinthub",
"filename": "fingerprinthub_web.json",
"model": FingerPrintHubFingerprint,
"service": FingerPrintHubFingerprintService,
"data_key": None, # FingerPrintHub 是数组格式
"file_format": "json",
},
{
"type": "arl",
"filename": "ARL.yaml",
"model": ARLFingerprint,
"service": ARLFingerprintService,
"data_key": None, # ARL 是 YAML 数组格式
"file_format": "yaml",
},
]
@@ -68,6 +109,7 @@ class Command(BaseCommand):
model = item["model"]
service_class = item["service"]
data_key = item["data_key"]
file_format = item.get("file_format", "json")
# 检查数据库是否已有数据
existing_count = model.objects.count()
@@ -87,11 +129,14 @@ class Command(BaseCommand):
failed += 1
continue
# 读取并解析 JSON
# 读取并解析文件(支持 JSON 和 YAML
try:
with open(src_path, "r", encoding="utf-8") as f:
json_data = json.load(f)
except (json.JSONDecodeError, OSError) as exc:
if file_format == "yaml":
file_data = yaml.safe_load(f)
else:
file_data = json.load(f)
except (json.JSONDecodeError, yaml.YAMLError, OSError) as exc:
self.stdout.write(self.style.ERROR(
f"[{fp_type}] 读取指纹文件失败: {exc}"
))
@@ -99,7 +144,7 @@ class Command(BaseCommand):
continue
# 提取指纹数据(根据不同格式处理)
fingerprints = self._extract_fingerprints(json_data, data_key, fp_type)
fingerprints = self._extract_fingerprints(file_data, data_key, fp_type)
if not fingerprints:
self.stdout.write(self.style.WARNING(
f"[{fp_type}] 指纹文件中没有有效数据,跳过"

View File

@@ -4,7 +4,14 @@
"""
from .engine import WorkerNode, ScanEngine, Wordlist, NucleiTemplateRepo
from .fingerprints import EholeFingerprint, GobyFingerprint, WappalyzerFingerprint
from .fingerprints import (
EholeFingerprint,
GobyFingerprint,
WappalyzerFingerprint,
FingersFingerprint,
FingerPrintHubFingerprint,
ARLFingerprint,
)
__all__ = [
# 核心 Models
@@ -16,4 +23,7 @@ __all__ = [
"EholeFingerprint",
"GobyFingerprint",
"WappalyzerFingerprint",
"FingersFingerprint",
"FingerPrintHubFingerprint",
"ARLFingerprint",
]

View File

@@ -106,3 +106,90 @@ class WappalyzerFingerprint(models.Model):
def __str__(self) -> str:
return f"{self.name}"
class FingersFingerprint(models.Model):
"""Fingers 格式指纹规则 (fingers_http.json)
使用正则表达式和标签进行匹配,支持 favicon hash、header、body 等多种检测方式
"""
name = models.CharField(max_length=300, unique=True, help_text='指纹名称')
link = models.URLField(max_length=500, blank=True, default='', help_text='相关链接')
rule = models.JSONField(default=list, help_text='匹配规则数组')
tag = models.JSONField(default=list, help_text='标签数组')
focus = models.BooleanField(default=False, help_text='是否重点关注')
default_port = models.JSONField(default=list, blank=True, help_text='默认端口数组')
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
db_table = 'fingers_fingerprint'
verbose_name = 'Fingers 指纹'
verbose_name_plural = 'Fingers 指纹'
ordering = ['-created_at']
indexes = [
models.Index(fields=['name']),
models.Index(fields=['link']),
models.Index(fields=['focus']),
models.Index(fields=['-created_at']),
]
def __str__(self) -> str:
return f"{self.name}"
class FingerPrintHubFingerprint(models.Model):
"""FingerPrintHub 格式指纹规则 (fingerprinthub_web.json)
基于 nuclei 模板格式,使用 HTTP 请求和响应特征进行匹配
"""
fp_id = models.CharField(max_length=200, unique=True, help_text='指纹ID')
name = models.CharField(max_length=300, help_text='指纹名称')
author = models.CharField(max_length=200, blank=True, default='', help_text='作者')
tags = models.CharField(max_length=500, blank=True, default='', help_text='标签')
severity = models.CharField(max_length=50, blank=True, default='info', help_text='严重程度')
metadata = models.JSONField(default=dict, blank=True, help_text='元数据')
http = models.JSONField(default=list, help_text='HTTP 匹配规则')
source_file = models.CharField(max_length=500, blank=True, default='', help_text='来源文件')
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
db_table = 'fingerprinthub_fingerprint'
verbose_name = 'FingerPrintHub 指纹'
verbose_name_plural = 'FingerPrintHub 指纹'
ordering = ['-created_at']
indexes = [
models.Index(fields=['fp_id']),
models.Index(fields=['name']),
models.Index(fields=['author']),
models.Index(fields=['severity']),
models.Index(fields=['-created_at']),
]
def __str__(self) -> str:
return f"{self.name} ({self.fp_id})"
class ARLFingerprint(models.Model):
"""ARL 格式指纹规则 (ARL.yaml)
使用简单的 name + rule 表达式格式
"""
name = models.CharField(max_length=300, unique=True, help_text='指纹名称')
rule = models.TextField(help_text='匹配规则表达式')
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
db_table = 'arl_fingerprint'
verbose_name = 'ARL 指纹'
verbose_name_plural = 'ARL 指纹'
ordering = ['-created_at']
indexes = [
models.Index(fields=['name']),
models.Index(fields=['-created_at']),
]
def __str__(self) -> str:
return f"{self.name}"

View File

@@ -6,9 +6,15 @@
from .ehole import EholeFingerprintSerializer
from .goby import GobyFingerprintSerializer
from .wappalyzer import WappalyzerFingerprintSerializer
from .fingers import FingersFingerprintSerializer
from .fingerprinthub import FingerPrintHubFingerprintSerializer
from .arl import ARLFingerprintSerializer
__all__ = [
"EholeFingerprintSerializer",
"GobyFingerprintSerializer",
"WappalyzerFingerprintSerializer",
"FingersFingerprintSerializer",
"FingerPrintHubFingerprintSerializer",
"ARLFingerprintSerializer",
]

View File

@@ -0,0 +1,31 @@
"""ARL 指纹 Serializer"""
from rest_framework import serializers
from apps.engine.models import ARLFingerprint
class ARLFingerprintSerializer(serializers.ModelSerializer):
"""ARL 指纹序列化器
字段映射:
- name: 指纹名称 (必填, 唯一)
- rule: 匹配规则表达式 (必填)
"""
class Meta:
model = ARLFingerprint
fields = ['id', 'name', 'rule', 'created_at']
read_only_fields = ['id', 'created_at']
def validate_name(self, value):
"""校验 name 字段"""
if not value or not value.strip():
raise serializers.ValidationError("name 字段不能为空")
return value.strip()
def validate_rule(self, value):
"""校验 rule 字段"""
if not value or not value.strip():
raise serializers.ValidationError("rule 字段不能为空")
return value.strip()

View File

@@ -0,0 +1,50 @@
"""FingerPrintHub 指纹 Serializer"""
from rest_framework import serializers
from apps.engine.models import FingerPrintHubFingerprint
class FingerPrintHubFingerprintSerializer(serializers.ModelSerializer):
"""FingerPrintHub 指纹序列化器
字段映射:
- fp_id: 指纹ID (必填, 唯一)
- name: 指纹名称 (必填)
- author: 作者 (可选)
- tags: 标签字符串 (可选)
- severity: 严重程度 (可选, 默认 'info')
- metadata: 元数据 JSON (可选)
- http: HTTP 匹配规则数组 (必填)
- source_file: 来源文件 (可选)
"""
class Meta:
model = FingerPrintHubFingerprint
fields = ['id', 'fp_id', 'name', 'author', 'tags', 'severity',
'metadata', 'http', 'source_file', 'created_at']
read_only_fields = ['id', 'created_at']
def validate_fp_id(self, value):
"""校验 fp_id 字段"""
if not value or not value.strip():
raise serializers.ValidationError("fp_id 字段不能为空")
return value.strip()
def validate_name(self, value):
"""校验 name 字段"""
if not value or not value.strip():
raise serializers.ValidationError("name 字段不能为空")
return value.strip()
def validate_http(self, value):
"""校验 http 字段"""
if not isinstance(value, list):
raise serializers.ValidationError("http 必须是数组")
return value
def validate_metadata(self, value):
"""校验 metadata 字段"""
if not isinstance(value, dict):
raise serializers.ValidationError("metadata 必须是对象")
return value

View File

@@ -0,0 +1,48 @@
"""Fingers 指纹 Serializer"""
from rest_framework import serializers
from apps.engine.models import FingersFingerprint
class FingersFingerprintSerializer(serializers.ModelSerializer):
"""Fingers 指纹序列化器
字段映射:
- name: 指纹名称 (必填, 唯一)
- link: 相关链接 (可选)
- rule: 匹配规则数组 (必填)
- tag: 标签数组 (可选)
- focus: 是否重点关注 (可选, 默认 False)
- default_port: 默认端口数组 (可选)
"""
class Meta:
model = FingersFingerprint
fields = ['id', 'name', 'link', 'rule', 'tag', 'focus',
'default_port', 'created_at']
read_only_fields = ['id', 'created_at']
def validate_name(self, value):
"""校验 name 字段"""
if not value or not value.strip():
raise serializers.ValidationError("name 字段不能为空")
return value.strip()
def validate_rule(self, value):
"""校验 rule 字段"""
if not isinstance(value, list):
raise serializers.ValidationError("rule 必须是数组")
return value
def validate_tag(self, value):
"""校验 tag 字段"""
if not isinstance(value, list):
raise serializers.ValidationError("tag 必须是数组")
return value
def validate_default_port(self, value):
"""校验 default_port 字段"""
if not isinstance(value, list):
raise serializers.ValidationError("default_port 必须是数组")
return value

View File

@@ -66,6 +66,7 @@ def get_start_agent_script(
# 替换变量
script = script.replace("{{HEARTBEAT_API_URL}}", heartbeat_api_url or '')
script = script.replace("{{WORKER_ID}}", str(worker_id) if worker_id else '')
script = script.replace("{{WORKER_API_KEY}}", getattr(settings, 'WORKER_API_KEY', ''))
# 注入镜像版本配置(确保远程节点使用相同版本)
docker_user = getattr(settings, 'DOCKER_USER', 'yyhuni')

View File

@@ -7,10 +7,16 @@ from .base import BaseFingerprintService
from .ehole import EholeFingerprintService
from .goby import GobyFingerprintService
from .wappalyzer import WappalyzerFingerprintService
from .fingers_service import FingersFingerprintService
from .fingerprinthub_service import FingerPrintHubFingerprintService
from .arl_service import ARLFingerprintService
__all__ = [
"BaseFingerprintService",
"EholeFingerprintService",
"GobyFingerprintService",
"WappalyzerFingerprintService",
"FingersFingerprintService",
"FingerPrintHubFingerprintService",
"ARLFingerprintService",
]

View File

@@ -0,0 +1,110 @@
"""ARL 指纹管理 Service
实现 ARL 格式指纹的校验、转换和导出逻辑
支持 YAML 格式的导入导出
"""
import logging
import yaml
from apps.engine.models import ARLFingerprint
from .base import BaseFingerprintService
logger = logging.getLogger(__name__)
class ARLFingerprintService(BaseFingerprintService):
"""ARL 指纹管理服务(继承基类,实现 ARL 特定逻辑)"""
model = ARLFingerprint
def validate_fingerprint(self, item: dict) -> bool:
"""
校验单条 ARL 指纹
校验规则:
- name 字段必须存在且非空
- rule 字段必须存在且非空
Args:
item: 单条指纹数据
Returns:
bool: 是否有效
"""
name = item.get('name', '')
rule = item.get('rule', '')
return bool(name and str(name).strip()) and bool(rule and str(rule).strip())
def to_model_data(self, item: dict) -> dict:
"""
转换 ARL YAML 格式为 Model 字段
Args:
item: 原始 ARL YAML 数据
Returns:
dict: Model 字段数据
"""
return {
'name': str(item.get('name', '')).strip(),
'rule': str(item.get('rule', '')).strip(),
}
def get_export_data(self) -> list:
"""
获取导出数据ARL 格式 - 数组,用于 YAML 导出)
Returns:
list: ARL 格式的数据(数组格式)
[
{"name": "...", "rule": "..."},
...
]
"""
fingerprints = self.model.objects.all()
return [
{
'name': fp.name,
'rule': fp.rule,
}
for fp in fingerprints
]
def export_to_yaml(self, output_path: str) -> int:
"""
导出所有指纹到 YAML 文件
Args:
output_path: 输出文件路径
Returns:
int: 导出的指纹数量
"""
data = self.get_export_data()
with open(output_path, 'w', encoding='utf-8') as f:
yaml.dump(data, f, allow_unicode=True, default_flow_style=False, sort_keys=False)
count = len(data)
logger.info("导出 ARL 指纹文件: %s, 数量: %d", output_path, count)
return count
def parse_yaml_import(self, yaml_content: str) -> list:
"""
解析 YAML 格式的导入内容
Args:
yaml_content: YAML 格式的字符串内容
Returns:
list: 解析后的指纹数据列表
Raises:
ValueError: 当 YAML 格式无效时
"""
try:
data = yaml.safe_load(yaml_content)
if not isinstance(data, list):
raise ValueError("ARL YAML 文件必须是数组格式")
return data
except yaml.YAMLError as e:
raise ValueError(f"无效的 YAML 格式: {e}")

View File

@@ -0,0 +1,110 @@
"""FingerPrintHub 指纹管理 Service
实现 FingerPrintHub 格式指纹的校验、转换和导出逻辑
"""
from apps.engine.models import FingerPrintHubFingerprint
from .base import BaseFingerprintService
class FingerPrintHubFingerprintService(BaseFingerprintService):
"""FingerPrintHub 指纹管理服务(继承基类,实现 FingerPrintHub 特定逻辑)"""
model = FingerPrintHubFingerprint
def validate_fingerprint(self, item: dict) -> bool:
"""
校验单条 FingerPrintHub 指纹
校验规则:
- id 字段必须存在且非空
- info 字段必须存在且包含 name
- http 字段必须是数组
Args:
item: 单条指纹数据
Returns:
bool: 是否有效
"""
fp_id = item.get('id', '')
info = item.get('info', {})
http = item.get('http')
if not fp_id or not str(fp_id).strip():
return False
if not isinstance(info, dict) or not info.get('name'):
return False
if not isinstance(http, list):
return False
return True
def to_model_data(self, item: dict) -> dict:
"""
转换 FingerPrintHub JSON 格式为 Model 字段
字段映射(嵌套结构转扁平):
- id (JSON) → fp_id (Model)
- info.name (JSON) → name (Model)
- info.author (JSON) → author (Model)
- info.tags (JSON) → tags (Model)
- info.severity (JSON) → severity (Model)
- info.metadata (JSON) → metadata (Model)
- http (JSON) → http (Model)
- _source_file (JSON) → source_file (Model)
Args:
item: 原始 FingerPrintHub JSON 数据
Returns:
dict: Model 字段数据
"""
info = item.get('info', {})
return {
'fp_id': str(item.get('id', '')).strip(),
'name': str(info.get('name', '')).strip(),
'author': info.get('author', ''),
'tags': info.get('tags', ''),
'severity': info.get('severity', 'info'),
'metadata': info.get('metadata', {}),
'http': item.get('http', []),
'source_file': item.get('_source_file', ''),
}
def get_export_data(self) -> list:
"""
获取导出数据FingerPrintHub JSON 格式 - 数组)
Returns:
list: FingerPrintHub 格式的 JSON 数据(数组格式)
[
{
"id": "...",
"info": {"name": "...", "author": "...", "tags": "...",
"severity": "...", "metadata": {...}},
"http": [...],
"_source_file": "..."
},
...
]
"""
fingerprints = self.model.objects.all()
data = []
for fp in fingerprints:
item = {
'id': fp.fp_id,
'info': {
'name': fp.name,
'author': fp.author,
'tags': fp.tags,
'severity': fp.severity,
'metadata': fp.metadata,
},
'http': fp.http,
}
# 只有当 source_file 非空时才添加该字段
if fp.source_file:
item['_source_file'] = fp.source_file
data.append(item)
return data

View File

@@ -0,0 +1,83 @@
"""Fingers 指纹管理 Service
实现 Fingers 格式指纹的校验、转换和导出逻辑
"""
from apps.engine.models import FingersFingerprint
from .base import BaseFingerprintService
class FingersFingerprintService(BaseFingerprintService):
"""Fingers 指纹管理服务(继承基类,实现 Fingers 特定逻辑)"""
model = FingersFingerprint
def validate_fingerprint(self, item: dict) -> bool:
"""
校验单条 Fingers 指纹
校验规则:
- name 字段必须存在且非空
- rule 字段必须是数组
Args:
item: 单条指纹数据
Returns:
bool: 是否有效
"""
name = item.get('name', '')
rule = item.get('rule')
return bool(name and str(name).strip()) and isinstance(rule, list)
def to_model_data(self, item: dict) -> dict:
"""
转换 Fingers JSON 格式为 Model 字段
字段映射:
- default_port (JSON) → default_port (Model)
Args:
item: 原始 Fingers JSON 数据
Returns:
dict: Model 字段数据
"""
return {
'name': str(item.get('name', '')).strip(),
'link': item.get('link', ''),
'rule': item.get('rule', []),
'tag': item.get('tag', []),
'focus': item.get('focus', False),
'default_port': item.get('default_port', []),
}
def get_export_data(self) -> list:
"""
获取导出数据Fingers JSON 格式 - 数组)
Returns:
list: Fingers 格式的 JSON 数据(数组格式)
[
{"name": "...", "link": "...", "rule": [...], "tag": [...],
"focus": false, "default_port": [...]},
...
]
"""
fingerprints = self.model.objects.all()
data = []
for fp in fingerprints:
item = {
'name': fp.name,
'link': fp.link,
'rule': fp.rule,
'tag': fp.tag,
}
# 只有当 focus 为 True 时才添加该字段(保持与原始格式一致)
if fp.focus:
item['focus'] = fp.focus
# 只有当 default_port 非空时才添加该字段
if fp.default_port:
item['default_port'] = fp.default_port
data.append(item)
return data

View File

@@ -186,7 +186,6 @@ class NucleiTemplateRepoService:
RuntimeError: Git 命令执行失败
"""
import subprocess
from apps.common.utils.git_proxy import get_git_proxy_url
obj = self._get_repo_obj(repo_id)
@@ -197,14 +196,12 @@ class NucleiTemplateRepoService:
cmd: List[str]
action: str
# 获取代理后的 URL如果启用了 Git 加速)
proxied_url = get_git_proxy_url(obj.repo_url)
if proxied_url != obj.repo_url:
logger.info("使用 Git 加速: %s -> %s", obj.repo_url, proxied_url)
# 直接使用原始 URL不再使用 Git 加速)
repo_url = obj.repo_url
# 判断是 clone 还是 pull
if git_dir.is_dir():
# 检查远程地址是否变化(比较原始 URL不是代理 URL
# 检查远程地址是否变化
current_remote = subprocess.run(
["git", "-C", str(local_path), "remote", "get-url", "origin"],
check=False,
@@ -214,13 +211,13 @@ class NucleiTemplateRepoService:
)
current_url = current_remote.stdout.strip() if current_remote.returncode == 0 else ""
# 检查是否需要重新 clone(原始 URL 或代理 URL 变化都需要)
if current_url not in [obj.repo_url, proxied_url]:
# 检查是否需要重新 clone
if current_url != repo_url:
# 远程地址变化,删除旧目录重新 clone
logger.info("nuclei 模板仓库 %s 远程地址变化,重新 clone: %s -> %s", obj.id, current_url, obj.repo_url)
logger.info("nuclei 模板仓库 %s 远程地址变化,重新 clone: %s -> %s", obj.id, current_url, repo_url)
shutil.rmtree(local_path)
local_path.mkdir(parents=True, exist_ok=True)
cmd = ["git", "clone", "--depth", "1", proxied_url, str(local_path)]
cmd = ["git", "clone", "--depth", "1", repo_url, str(local_path)]
action = "clone"
else:
# 已有仓库且地址未变,执行 pull
@@ -231,7 +228,7 @@ class NucleiTemplateRepoService:
if local_path.exists() and not local_path.is_dir():
raise RuntimeError(f"本地路径已存在且不是目录: {local_path}")
# --depth 1 浅克隆,只获取最新提交,节省空间和时间
cmd = ["git", "clone", "--depth", "1", proxied_url, str(local_path)]
cmd = ["git", "clone", "--depth", "1", repo_url, str(local_path)]
action = "clone"
# 执行 Git 命令

View File

@@ -274,7 +274,7 @@ class TaskDistributor:
network_arg = ""
server_url = f"https://{settings.PUBLIC_HOST}:{settings.PUBLIC_PORT}"
# 挂载路径(统一挂载 /opt/xingrin
# 挂载路径(统一挂载 /opt/xingrin,扫描工具在 /opt/xingrin-tools/bin 不受影响
host_xingrin_dir = "/opt/xingrin"
# 环境变量SERVER_URL + IS_LOCAL其他配置容器启动时从配置中心获取
@@ -284,6 +284,7 @@ class TaskDistributor:
env_vars = [
f"-e SERVER_URL={shlex.quote(server_url)}",
f"-e IS_LOCAL={is_local_str}",
f"-e WORKER_API_KEY={shlex.quote(settings.WORKER_API_KEY)}", # Worker API 认证密钥
"-e PREFECT_HOME=/tmp/.prefect", # 设置 Prefect 数据目录到可写位置
"-e PREFECT_SERVER_EPHEMERAL_ENABLED=true", # 启用 ephemeral server本地临时服务器
"-e PREFECT_SERVER_EPHEMERAL_STARTUP_TIMEOUT_SECONDS=120", # 增加启动超时时间
@@ -311,11 +312,10 @@ class TaskDistributor:
# - 本地 Workerinstall.sh 已预拉取镜像,直接使用本地版本
# - 远程 Workerdeploy 时已预拉取镜像,直接使用本地版本
# - 避免每次任务都检查 Docker Hub提升性能和稳定性
# 使用双引号包裹 sh -c 命令,内部 shlex.quote 生成的单引号参数可正确解析
cmd = f'''docker run --rm -d --pull=missing {network_arg} \
{' '.join(env_vars)} \
{' '.join(volumes)} \
{self.docker_image} \
cmd = f'''docker run --rm -d --pull=missing {network_arg} \\
{' '.join(env_vars)} \\
{' '.join(volumes)} \\
{self.docker_image} \\
sh -c "{inner_cmd}"'''
return cmd

View File

@@ -11,6 +11,9 @@ from .views.fingerprints import (
EholeFingerprintViewSet,
GobyFingerprintViewSet,
WappalyzerFingerprintViewSet,
FingersFingerprintViewSet,
FingerPrintHubFingerprintViewSet,
ARLFingerprintViewSet,
)
@@ -24,6 +27,9 @@ router.register(r"nuclei/repos", NucleiTemplateRepoViewSet, basename="nuclei-rep
router.register(r"fingerprints/ehole", EholeFingerprintViewSet, basename="ehole-fingerprint")
router.register(r"fingerprints/goby", GobyFingerprintViewSet, basename="goby-fingerprint")
router.register(r"fingerprints/wappalyzer", WappalyzerFingerprintViewSet, basename="wappalyzer-fingerprint")
router.register(r"fingerprints/fingers", FingersFingerprintViewSet, basename="fingers-fingerprint")
router.register(r"fingerprints/fingerprinthub", FingerPrintHubFingerprintViewSet, basename="fingerprinthub-fingerprint")
router.register(r"fingerprints/arl", ARLFingerprintViewSet, basename="arl-fingerprint")
urlpatterns = [
path("", include(router.urls)),

View File

@@ -7,10 +7,16 @@ from .base import BaseFingerprintViewSet
from .ehole import EholeFingerprintViewSet
from .goby import GobyFingerprintViewSet
from .wappalyzer import WappalyzerFingerprintViewSet
from .fingers import FingersFingerprintViewSet
from .fingerprinthub import FingerPrintHubFingerprintViewSet
from .arl import ARLFingerprintViewSet
__all__ = [
"BaseFingerprintViewSet",
"EholeFingerprintViewSet",
"GobyFingerprintViewSet",
"WappalyzerFingerprintViewSet",
"FingersFingerprintViewSet",
"FingerPrintHubFingerprintViewSet",
"ARLFingerprintViewSet",
]

View File

@@ -0,0 +1,122 @@
"""ARL 指纹管理 ViewSet"""
import yaml
from django.http import HttpResponse
from rest_framework.decorators import action
from rest_framework.exceptions import ValidationError
from apps.common.pagination import BasePagination
from apps.common.response_helpers import success_response
from apps.engine.models import ARLFingerprint
from apps.engine.serializers.fingerprints import ARLFingerprintSerializer
from apps.engine.services.fingerprints import ARLFingerprintService
from .base import BaseFingerprintViewSet
class ARLFingerprintViewSet(BaseFingerprintViewSet):
"""ARL 指纹管理 ViewSet
继承自 BaseFingerprintViewSet提供以下 API
标准 CRUDModelViewSet
- GET / 列表查询(分页)
- POST / 创建单条
- GET /{id}/ 获取详情
- PUT /{id}/ 更新
- DELETE /{id}/ 删除
批量操作(继承自基类):
- POST /batch_create/ 批量创建JSON body
- POST /import_file/ 文件导入multipart/form-data支持 YAML
- POST /bulk-delete/ 批量删除
- POST /delete-all/ 删除所有
- GET /export/ 导出下载YAML 格式)
智能过滤语法filter 参数):
- name="word" 模糊匹配 name 字段
- name=="WordPress" 精确匹配
- rule="body=" 按规则内容筛选
"""
queryset = ARLFingerprint.objects.all()
serializer_class = ARLFingerprintSerializer
pagination_class = BasePagination
service_class = ARLFingerprintService
# 排序配置
ordering_fields = ['created_at', 'name']
ordering = ['-created_at']
# ARL 过滤字段映射
FILTER_FIELD_MAPPING = {
'name': 'name',
'rule': 'rule',
}
def parse_import_data(self, json_data) -> list:
"""
解析 ARL 格式的导入数据JSON 格式)
输入格式:[{...}, {...}] 数组格式
返回:指纹列表
"""
if isinstance(json_data, list):
return json_data
return []
def get_export_filename(self) -> str:
"""导出文件名"""
return 'ARL.yaml'
@action(detail=False, methods=['post'])
def import_file(self, request):
"""
文件导入(支持 YAML 和 JSON 格式)
POST /api/engine/fingerprints/arl/import_file/
请求格式multipart/form-data
- file: YAML 或 JSON 文件
返回:同 batch_create
"""
file = request.FILES.get('file')
if not file:
raise ValidationError('缺少文件')
filename = file.name.lower()
content = file.read().decode('utf-8')
try:
if filename.endswith('.yaml') or filename.endswith('.yml'):
# YAML 格式
fingerprints = yaml.safe_load(content)
else:
# JSON 格式
import json
fingerprints = json.loads(content)
except (yaml.YAMLError, json.JSONDecodeError) as e:
raise ValidationError(f'无效的文件格式: {e}')
if not isinstance(fingerprints, list):
raise ValidationError('文件内容必须是数组格式')
if not fingerprints:
raise ValidationError('文件中没有有效的指纹数据')
result = self.get_service().batch_create_fingerprints(fingerprints)
return success_response(data=result)
@action(detail=False, methods=['get'])
def export(self, request):
"""
导出指纹YAML 格式)
GET /api/engine/fingerprints/arl/export/
返回YAML 文件下载
"""
data = self.get_service().get_export_data()
content = yaml.dump(data, allow_unicode=True, default_flow_style=False, sort_keys=False)
response = HttpResponse(content, content_type='application/x-yaml')
response['Content-Disposition'] = f'attachment; filename="{self.get_export_filename()}"'
return response

View File

@@ -13,6 +13,7 @@ from rest_framework.response import Response
from rest_framework.exceptions import ValidationError
from apps.common.pagination import BasePagination
from apps.common.response_helpers import success_response
from apps.common.utils.filter_utils import apply_filters
logger = logging.getLogger(__name__)
@@ -129,7 +130,7 @@ class BaseFingerprintViewSet(viewsets.ModelViewSet):
raise ValidationError('fingerprints 必须是数组')
result = self.get_service().batch_create_fingerprints(fingerprints)
return Response(result, status=status.HTTP_201_CREATED)
return success_response(data=result, status_code=status.HTTP_201_CREATED)
@action(detail=False, methods=['post'])
def import_file(self, request):
@@ -156,7 +157,7 @@ class BaseFingerprintViewSet(viewsets.ModelViewSet):
raise ValidationError('文件中没有有效的指纹数据')
result = self.get_service().batch_create_fingerprints(fingerprints)
return Response(result, status=status.HTTP_201_CREATED)
return success_response(data=result, status_code=status.HTTP_201_CREATED)
@action(detail=False, methods=['post'], url_path='bulk-delete')
def bulk_delete(self, request):
@@ -174,7 +175,7 @@ class BaseFingerprintViewSet(viewsets.ModelViewSet):
raise ValidationError('ids 必须是数组')
deleted_count = self.queryset.model.objects.filter(id__in=ids).delete()[0]
return Response({'deleted': deleted_count})
return success_response(data={'deleted': deleted_count})
@action(detail=False, methods=['post'], url_path='delete-all')
def delete_all(self, request):
@@ -185,7 +186,7 @@ class BaseFingerprintViewSet(viewsets.ModelViewSet):
返回:{"deleted": 1000}
"""
deleted_count = self.queryset.model.objects.all().delete()[0]
return Response({'deleted': deleted_count})
return success_response(data={'deleted': deleted_count})
@action(detail=False, methods=['get'])
def export(self, request):

View File

@@ -0,0 +1,73 @@
"""FingerPrintHub 指纹管理 ViewSet"""
from apps.common.pagination import BasePagination
from apps.engine.models import FingerPrintHubFingerprint
from apps.engine.serializers.fingerprints import FingerPrintHubFingerprintSerializer
from apps.engine.services.fingerprints import FingerPrintHubFingerprintService
from .base import BaseFingerprintViewSet
class FingerPrintHubFingerprintViewSet(BaseFingerprintViewSet):
"""FingerPrintHub 指纹管理 ViewSet
继承自 BaseFingerprintViewSet提供以下 API
标准 CRUDModelViewSet
- GET / 列表查询(分页)
- POST / 创建单条
- GET /{id}/ 获取详情
- PUT /{id}/ 更新
- DELETE /{id}/ 删除
批量操作(继承自基类):
- POST /batch_create/ 批量创建JSON body
- POST /import_file/ 文件导入multipart/form-data
- POST /bulk-delete/ 批量删除
- POST /delete-all/ 删除所有
- GET /export/ 导出下载
智能过滤语法filter 参数):
- name="word" 模糊匹配 name 字段
- fp_id=="xxx" 精确匹配指纹ID
- author="xxx" 按作者筛选
- severity="info" 按严重程度筛选
- tags="cms" 按标签筛选
"""
queryset = FingerPrintHubFingerprint.objects.all()
serializer_class = FingerPrintHubFingerprintSerializer
pagination_class = BasePagination
service_class = FingerPrintHubFingerprintService
# 排序配置
ordering_fields = ['created_at', 'name', 'severity']
ordering = ['-created_at']
# FingerPrintHub 过滤字段映射
FILTER_FIELD_MAPPING = {
'fp_id': 'fp_id',
'name': 'name',
'author': 'author',
'tags': 'tags',
'severity': 'severity',
'source_file': 'source_file',
}
# JSON 数组字段(使用 __contains 查询)
JSON_ARRAY_FIELDS = ['http']
def parse_import_data(self, json_data) -> list:
"""
解析 FingerPrintHub JSON 格式的导入数据
输入格式:[{...}, {...}] 数组格式
返回:指纹列表
"""
if isinstance(json_data, list):
return json_data
return []
def get_export_filename(self) -> str:
"""导出文件名"""
return 'fingerprinthub_web.json'

View File

@@ -0,0 +1,69 @@
"""Fingers 指纹管理 ViewSet"""
from apps.common.pagination import BasePagination
from apps.engine.models import FingersFingerprint
from apps.engine.serializers.fingerprints import FingersFingerprintSerializer
from apps.engine.services.fingerprints import FingersFingerprintService
from .base import BaseFingerprintViewSet
class FingersFingerprintViewSet(BaseFingerprintViewSet):
"""Fingers 指纹管理 ViewSet
继承自 BaseFingerprintViewSet提供以下 API
标准 CRUDModelViewSet
- GET / 列表查询(分页)
- POST / 创建单条
- GET /{id}/ 获取详情
- PUT /{id}/ 更新
- DELETE /{id}/ 删除
批量操作(继承自基类):
- POST /batch_create/ 批量创建JSON body
- POST /import_file/ 文件导入multipart/form-data
- POST /bulk-delete/ 批量删除
- POST /delete-all/ 删除所有
- GET /export/ 导出下载
智能过滤语法filter 参数):
- name="word" 模糊匹配 name 字段
- name=="WordPress" 精确匹配
- tag="cms" 按标签筛选
- focus="true" 按重点关注筛选
"""
queryset = FingersFingerprint.objects.all()
serializer_class = FingersFingerprintSerializer
pagination_class = BasePagination
service_class = FingersFingerprintService
# 排序配置
ordering_fields = ['created_at', 'name']
ordering = ['-created_at']
# Fingers 过滤字段映射
FILTER_FIELD_MAPPING = {
'name': 'name',
'link': 'link',
'focus': 'focus',
}
# JSON 数组字段(使用 __contains 查询)
JSON_ARRAY_FIELDS = ['tag', 'rule', 'default_port']
def parse_import_data(self, json_data) -> list:
"""
解析 Fingers JSON 格式的导入数据
输入格式:[{...}, {...}] 数组格式
返回:指纹列表
"""
if isinstance(json_data, list):
return json_data
return []
def get_export_filename(self) -> str:
"""导出文件名"""
return 'fingers_http.json'

View File

@@ -31,6 +31,8 @@ from rest_framework.decorators import action
from rest_framework.request import Request
from rest_framework.response import Response
from apps.common.response_helpers import success_response, error_response
from apps.common.error_codes import ErrorCodes
from apps.engine.models import NucleiTemplateRepo
from apps.engine.serializers import NucleiTemplateRepoSerializer
from apps.engine.services import NucleiTemplateRepoService
@@ -107,18 +109,30 @@ class NucleiTemplateRepoViewSet(viewsets.ModelViewSet):
try:
repo_id = int(pk) if pk is not None else None
except (TypeError, ValueError):
return Response({"message": "无效的仓库 ID"}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Invalid repository ID',
status_code=status.HTTP_400_BAD_REQUEST
)
# 调用 Service 层
try:
result = self.service.refresh_repo(repo_id)
except ValidationError as exc:
return Response({"message": str(exc)}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message=str(exc),
status_code=status.HTTP_400_BAD_REQUEST
)
except Exception as exc: # noqa: BLE001
logger.error("刷新 Nuclei 模板仓库失败: %s", exc, exc_info=True)
return Response({"message": f"刷新仓库失败: {exc}"}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
return error_response(
code=ErrorCodes.SERVER_ERROR,
message=f'Refresh failed: {exc}',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
return Response({"message": "刷新成功", "result": result}, status=status.HTTP_200_OK)
return success_response(data={'result': result})
# ==================== 自定义 Action: 模板只读浏览 ====================
@@ -142,18 +156,30 @@ class NucleiTemplateRepoViewSet(viewsets.ModelViewSet):
try:
repo_id = int(pk) if pk is not None else None
except (TypeError, ValueError):
return Response({"message": "无效的仓库 ID"}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Invalid repository ID',
status_code=status.HTTP_400_BAD_REQUEST
)
# 调用 Service 层,仅从当前本地目录读取目录树
try:
roots = self.service.get_template_tree(repo_id)
except ValidationError as exc:
return Response({"message": str(exc)}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message=str(exc),
status_code=status.HTTP_400_BAD_REQUEST
)
except Exception as exc: # noqa: BLE001
logger.error("获取 Nuclei 模板目录树失败: %s", exc, exc_info=True)
return Response({"message": "获取模板目录树失败"}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Failed to get template tree',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
return Response({"roots": roots})
return success_response(data={'roots': roots})
@action(detail=True, methods=["get"], url_path="templates/content")
def templates_content(self, request: Request, pk: str | None = None) -> Response:
@@ -174,23 +200,43 @@ class NucleiTemplateRepoViewSet(viewsets.ModelViewSet):
try:
repo_id = int(pk) if pk is not None else None
except (TypeError, ValueError):
return Response({"message": "无效的仓库 ID"}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Invalid repository ID',
status_code=status.HTTP_400_BAD_REQUEST
)
# 解析 path 参数
rel_path = (request.query_params.get("path", "") or "").strip()
if not rel_path:
return Response({"message": "缺少 path 参数"}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Missing path parameter',
status_code=status.HTTP_400_BAD_REQUEST
)
# 调用 Service 层
try:
result = self.service.get_template_content(repo_id, rel_path)
except ValidationError as exc:
return Response({"message": str(exc)}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message=str(exc),
status_code=status.HTTP_400_BAD_REQUEST
)
except Exception as exc: # noqa: BLE001
logger.error("获取 Nuclei 模板内容失败: %s", exc, exc_info=True)
return Response({"message": "获取模板内容失败"}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Failed to get template content',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
# 文件不存在
if result is None:
return Response({"message": "模板不存在或无法读取"}, status=status.HTTP_404_NOT_FOUND)
return Response(result)
return error_response(
code=ErrorCodes.NOT_FOUND,
message='Template not found or unreadable',
status_code=status.HTTP_404_NOT_FOUND
)
return success_response(data=result)

View File

@@ -9,6 +9,8 @@ from rest_framework.decorators import action
from rest_framework.response import Response
from apps.common.pagination import BasePagination
from apps.common.response_helpers import success_response, error_response
from apps.common.error_codes import ErrorCodes
from apps.engine.serializers.wordlist_serializer import WordlistSerializer
from apps.engine.services.wordlist_service import WordlistService
@@ -46,7 +48,11 @@ class WordlistViewSet(viewsets.ViewSet):
uploaded_file = request.FILES.get("file")
if not uploaded_file:
return Response({"error": "缺少字典文件"}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Missing wordlist file',
status_code=status.HTTP_400_BAD_REQUEST
)
try:
wordlist = self.service.create_wordlist(
@@ -55,21 +61,32 @@ class WordlistViewSet(viewsets.ViewSet):
uploaded_file=uploaded_file,
)
except ValidationError as exc:
return Response({"error": str(exc)}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message=str(exc),
status_code=status.HTTP_400_BAD_REQUEST
)
serializer = WordlistSerializer(wordlist)
return Response(serializer.data, status=status.HTTP_201_CREATED)
return success_response(data=serializer.data, status_code=status.HTTP_201_CREATED)
def destroy(self, request, pk=None):
"""删除字典记录"""
try:
wordlist_id = int(pk)
except (TypeError, ValueError):
return Response({"error": "无效的 ID"}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Invalid ID',
status_code=status.HTTP_400_BAD_REQUEST
)
success = self.service.delete_wordlist(wordlist_id)
if not success:
return Response({"error": "字典不存在"}, status=status.HTTP_404_NOT_FOUND)
return error_response(
code=ErrorCodes.NOT_FOUND,
status_code=status.HTTP_404_NOT_FOUND
)
return Response(status=status.HTTP_204_NO_CONTENT)
@@ -82,15 +99,27 @@ class WordlistViewSet(viewsets.ViewSet):
"""
name = (request.query_params.get("wordlist", "") or "").strip()
if not name:
return Response({"error": "缺少参数 wordlist"}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Missing parameter: wordlist',
status_code=status.HTTP_400_BAD_REQUEST
)
wordlist = self.service.get_wordlist_by_name(name)
if not wordlist:
return Response({"error": "字典不存在"}, status=status.HTTP_404_NOT_FOUND)
return error_response(
code=ErrorCodes.NOT_FOUND,
message='Wordlist not found',
status_code=status.HTTP_404_NOT_FOUND
)
file_path = wordlist.file_path
if not file_path or not os.path.exists(file_path):
return Response({"error": "字典文件不存在"}, status=status.HTTP_404_NOT_FOUND)
return error_response(
code=ErrorCodes.NOT_FOUND,
message='Wordlist file not found',
status_code=status.HTTP_404_NOT_FOUND
)
filename = os.path.basename(file_path)
response = FileResponse(open(file_path, "rb"), as_attachment=True, filename=filename)
@@ -106,22 +135,38 @@ class WordlistViewSet(viewsets.ViewSet):
try:
wordlist_id = int(pk)
except (TypeError, ValueError):
return Response({"error": "无效的 ID"}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Invalid ID',
status_code=status.HTTP_400_BAD_REQUEST
)
if request.method == "GET":
content = self.service.get_wordlist_content(wordlist_id)
if content is None:
return Response({"error": "字典不存在或文件无法读取"}, status=status.HTTP_404_NOT_FOUND)
return Response({"content": content})
return error_response(
code=ErrorCodes.NOT_FOUND,
message='Wordlist not found or file unreadable',
status_code=status.HTTP_404_NOT_FOUND
)
return success_response(data={"content": content})
elif request.method == "PUT":
content = request.data.get("content")
if content is None:
return Response({"error": "缺少 content 参数"}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Missing content parameter',
status_code=status.HTTP_400_BAD_REQUEST
)
wordlist = self.service.update_wordlist_content(wordlist_id, content)
if not wordlist:
return Response({"error": "字典不存在或更新失败"}, status=status.HTTP_404_NOT_FOUND)
return error_response(
code=ErrorCodes.NOT_FOUND,
message='Wordlist not found or update failed',
status_code=status.HTTP_404_NOT_FOUND
)
serializer = WordlistSerializer(wordlist)
return Response(serializer.data)
return success_response(data=serializer.data)

View File

@@ -9,6 +9,8 @@ from rest_framework import viewsets, status
from rest_framework.decorators import action
from rest_framework.response import Response
from apps.common.response_helpers import success_response, error_response
from apps.common.error_codes import ErrorCodes
from apps.engine.serializers import WorkerNodeSerializer
from apps.engine.services import WorkerService
from apps.common.signals import worker_delete_failed
@@ -111,9 +113,8 @@ class WorkerNodeViewSet(viewsets.ModelViewSet):
threading.Thread(target=_async_remote_uninstall, daemon=True).start()
# 3. 立即返回成功
return Response(
{"message": f"节点 {worker_name} 已删除"},
status=status.HTTP_200_OK
return success_response(
data={'name': worker_name}
)
@action(detail=True, methods=['post'])
@@ -190,11 +191,13 @@ class WorkerNodeViewSet(viewsets.ModelViewSet):
worker.status = 'online'
worker.save(update_fields=['status'])
return Response({
'status': 'ok',
'need_update': need_update,
'server_version': server_version
})
return success_response(
data={
'status': 'ok',
'needUpdate': need_update,
'serverVersion': server_version
}
)
def _trigger_remote_agent_update(self, worker, target_version: str):
"""
@@ -304,9 +307,10 @@ class WorkerNodeViewSet(viewsets.ModelViewSet):
is_local = request.data.get('is_local', True)
if not name:
return Response(
{'error': '缺少 name 参数'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Missing name parameter',
status_code=status.HTTP_400_BAD_REQUEST
)
worker, created = self.worker_service.register_worker(
@@ -314,11 +318,13 @@ class WorkerNodeViewSet(viewsets.ModelViewSet):
is_local=is_local
)
return Response({
'worker_id': worker.id,
'name': worker.name,
'created': created
})
return success_response(
data={
'workerId': worker.id,
'name': worker.name,
'created': created
}
)
@action(detail=False, methods=['get'])
def config(self, request):
@@ -334,13 +340,12 @@ class WorkerNodeViewSet(viewsets.ModelViewSet):
返回:
{
"db": {"host": "...", "port": "...", ...},
"redisUrl": "...",
"paths": {"results": "...", "logs": "..."}
}
配置逻辑:
- 本地 Worker (is_local=true): db_host=postgres, redis=redis:6379
- 远程 Worker (is_local=false): db_host=PUBLIC_HOST, redis=PUBLIC_HOST:6379
- 本地 Worker (is_local=true): db_host=postgres
- 远程 Worker (is_local=false): db_host=PUBLIC_HOST
"""
from django.conf import settings
import logging
@@ -365,39 +370,35 @@ class WorkerNodeViewSet(viewsets.ModelViewSet):
if is_local_worker:
# 本地 Worker直接用 Docker 内部服务名
worker_db_host = 'postgres'
worker_redis_url = 'redis://redis:6379/0'
else:
# 远程 Worker通过公网 IP 访问
public_host = settings.PUBLIC_HOST
if public_host in ('server', 'localhost', '127.0.0.1'):
logger.warning("远程 Worker 请求配置,但 PUBLIC_HOST=%s 不是有效的公网地址", public_host)
worker_db_host = public_host
worker_redis_url = f'redis://{public_host}:6379/0'
else:
# 远程数据库场景:所有 Worker 都用 DB_HOST
worker_db_host = db_host
worker_redis_url = getattr(settings, 'WORKER_REDIS_URL', 'redis://redis:6379/0')
logger.info("返回 Worker 配置 - db_host: %s, redis_url: %s", worker_db_host, worker_redis_url)
logger.info("返回 Worker 配置 - db_host: %s", worker_db_host)
return Response({
'db': {
'host': worker_db_host,
'port': str(settings.DATABASES['default']['PORT']),
'name': settings.DATABASES['default']['NAME'],
'user': settings.DATABASES['default']['USER'],
'password': settings.DATABASES['default']['PASSWORD'],
},
'redisUrl': worker_redis_url,
'paths': {
'results': getattr(settings, 'CONTAINER_RESULTS_MOUNT', '/opt/xingrin/results'),
'logs': getattr(settings, 'CONTAINER_LOGS_MOUNT', '/opt/xingrin/logs'),
},
'logging': {
'level': os.getenv('LOG_LEVEL', 'INFO'),
'enableCommandLogging': os.getenv('ENABLE_COMMAND_LOGGING', 'true').lower() == 'true',
},
'debug': settings.DEBUG,
# Git 加速配置(用于 Git clone 加速,如 Nuclei 模板仓库)
'gitMirror': settings.GIT_MIRROR,
})
return success_response(
data={
'db': {
'host': worker_db_host,
'port': str(settings.DATABASES['default']['PORT']),
'name': settings.DATABASES['default']['NAME'],
'user': settings.DATABASES['default']['USER'],
'password': settings.DATABASES['default']['PASSWORD'],
},
'paths': {
'results': getattr(settings, 'CONTAINER_RESULTS_MOUNT', '/opt/xingrin/results'),
'logs': getattr(settings, 'CONTAINER_LOGS_MOUNT', '/opt/xingrin/logs'),
},
'logging': {
'level': os.getenv('LOG_LEVEL', 'INFO'),
'enableCommandLogging': os.getenv('ENABLE_COMMAND_LOGGING', 'true').lower() == 'true',
},
'debug': settings.DEBUG,
}
)

View File

@@ -7,7 +7,7 @@
from django.conf import settings
# ==================== 路径配置 ====================
SCAN_TOOLS_BASE_PATH = getattr(settings, 'SCAN_TOOLS_BASE_PATH', '/opt/xingrin/tools')
SCAN_TOOLS_BASE_PATH = getattr(settings, 'SCAN_TOOLS_BASE_PATH', '/usr/local/bin')
# ==================== 子域名发现 ====================
@@ -35,7 +35,7 @@ SUBDOMAIN_DISCOVERY_COMMANDS = {
},
'sublist3r': {
'base': "python3 '{scan_tools_base}/Sublist3r/sublist3r.py' -d {domain} -o '{output_file}'",
'base': "python3 '/usr/local/share/Sublist3r/sublist3r.py' -d {domain} -o '{output_file}'",
'optional': {
'threads': '-t {threads}'
}
@@ -115,7 +115,7 @@ SITE_SCAN_COMMANDS = {
DIRECTORY_SCAN_COMMANDS = {
'ffuf': {
'base': "ffuf -u '{url}FUZZ' -se -ac -sf -json -w '{wordlist}'",
'base': "'{scan_tools_base}/ffuf' -u '{url}FUZZ' -se -ac -sf -json -w '{wordlist}'",
'optional': {
'delay': '-p {delay}',
'threads': '-t {threads}',
@@ -239,6 +239,9 @@ FINGERPRINT_DETECT_COMMANDS = {
'ehole': '--ehole {ehole}',
'goby': '--goby {goby}',
'wappalyzer': '--wappalyzer {wappalyzer}',
'fingers': '--fingers {fingers}',
'fingerprinthub': '--fingerprint {fingerprinthub}',
'arl': '--arl {arl}',
}
},
}

View File

@@ -53,7 +53,7 @@ subdomain_discovery:
resolve:
enabled: true
subdomain_resolve:
# timeout: auto # 自动根据候选子域数量计算
timeout: auto # 自动根据候选子域数量计算
# ==================== 端口扫描 ====================
port_scan:
@@ -87,7 +87,7 @@ fingerprint_detect:
tools:
xingfinger:
enabled: true
fingerprint-libs: [ehole, goby, wappalyzer] # 启用的指纹库ehole, goby, wappalyzer, fingers, fingerprinthub
fingerprint-libs: [ehole, goby, wappalyzer, fingers, fingerprinthub, arl] # 全部指纹库
# ==================== 目录扫描 ====================
directory_scan:

View File

@@ -37,28 +37,24 @@ logger = logging.getLogger(__name__)
def calculate_fingerprint_detect_timeout(
url_count: int,
base_per_url: float = 3.0,
min_timeout: int = 60
base_per_url: float = 10.0,
min_timeout: int = 300
) -> int:
"""
根据 URL 数量计算超时时间
公式:超时时间 = URL 数量 × 每 URL 基础时间
最小值:60秒
最小值:300秒
无上限
Args:
url_count: URL 数量
base_per_url: 每 URL 基础时间(秒),默认 3
min_timeout: 最小超时时间(秒),默认 60秒
base_per_url: 每 URL 基础时间(秒),默认 10
min_timeout: 最小超时时间(秒),默认 300秒
Returns:
int: 计算出的超时时间(秒)
示例:
100 URL × 3秒 = 300秒
1000 URL × 3秒 = 3000秒50分钟
10000 URL × 3秒 = 30000秒8.3小时)
"""
timeout = int(url_count * base_per_url)
return max(min_timeout, timeout)

View File

@@ -5,12 +5,13 @@ WebSocket Consumer - 通知实时推送
import json
import logging
import asyncio
from channels.generic.websocket import AsyncWebsocketConsumer
from apps.common.websocket_auth import AuthenticatedWebsocketConsumer
logger = logging.getLogger(__name__)
class NotificationConsumer(AsyncWebsocketConsumer):
class NotificationConsumer(AuthenticatedWebsocketConsumer):
"""
通知 WebSocket Consumer
@@ -23,9 +24,9 @@ class NotificationConsumer(AsyncWebsocketConsumer):
super().__init__(*args, **kwargs)
self.heartbeat_task = None # 心跳任务
async def connect(self):
async def on_connect(self):
"""
客户端连接时调用
客户端连接时调用(已通过认证)
加入通知广播组
"""
# 通知组名(所有客户端共享)

View File

@@ -305,6 +305,7 @@ def _push_via_api_callback(notification: Notification, server_url: str) -> None:
通过 HTTP 请求 Server 容器的 /api/callbacks/notification/ 接口。
Worker 无法直接访问 Redis需要由 Server 代为推送 WebSocket。
"""
import os
import requests
try:
@@ -318,8 +319,14 @@ def _push_via_api_callback(notification: Notification, server_url: str) -> None:
'created_at': notification.created_at.isoformat()
}
# 构建请求头(包含 Worker API Key
headers = {'Content-Type': 'application/json'}
worker_api_key = os.environ.get("WORKER_API_KEY", "")
if worker_api_key:
headers["X-Worker-API-Key"] = worker_api_key
# verify=False: 远程 Worker 回调 Server 时可能使用自签名证书
resp = requests.post(callback_url, json=data, timeout=5, verify=False)
resp = requests.post(callback_url, json=data, headers=headers, timeout=5, verify=False)
resp.raise_for_status()
logger.debug(f"通知回调推送成功 - ID: {notification.id}")

View File

@@ -7,13 +7,14 @@ from typing import Any
from django.http import JsonResponse
from django.utils import timezone
from rest_framework import status
from rest_framework.decorators import api_view, permission_classes
from rest_framework.permissions import AllowAny
from rest_framework.decorators import api_view
from rest_framework.request import Request
from rest_framework.response import Response
from rest_framework.views import APIView
from apps.common.pagination import BasePagination
from apps.common.response_helpers import success_response, error_response
from apps.common.error_codes import ErrorCodes
from .models import Notification
from .serializers import NotificationSerializer
from .types import NotificationLevel
@@ -60,34 +61,7 @@ def notifications_test(request):
}, status=500)
def build_api_response(
data: Any = None,
*,
message: str = '操作成功',
code: str = '200',
state: str = 'success',
status_code: int = status.HTTP_200_OK
) -> Response:
"""构建统一的 API 响应格式
Args:
data: 响应数据体(可选)
message: 响应消息
code: 响应代码
state: 响应状态success/error
status_code: HTTP 状态码
Returns:
DRF Response 对象
"""
payload = {
'code': code,
'state': state,
'message': message,
}
if data is not None:
payload['data'] = data
return Response(payload, status=status_code)
# build_api_response 已废弃,请使用 success_response/error_response
def _parse_bool(value: str | None) -> bool | None:
@@ -172,7 +146,7 @@ class NotificationUnreadCountView(APIView):
"""获取未读通知数量"""
service = NotificationService()
count = service.get_unread_count()
return build_api_response({'count': count}, message='获取未读数量成功')
return success_response(data={'count': count})
class NotificationMarkAllAsReadView(APIView):
@@ -192,7 +166,7 @@ class NotificationMarkAllAsReadView(APIView):
"""标记全部通知为已读"""
service = NotificationService()
updated = service.mark_all_as_read()
return build_api_response({'updated': updated}, message='全部标记已读成功')
return success_response(data={'updated': updated})
class NotificationSettingsView(APIView):
@@ -209,13 +183,13 @@ class NotificationSettingsView(APIView):
"""获取通知设置"""
service = NotificationSettingsService()
settings = service.get_settings()
return Response(settings)
return success_response(data=settings)
def put(self, request: Request) -> Response:
"""更新通知设置"""
service = NotificationSettingsService()
settings = service.update_settings(request.data)
return Response({'message': '已保存通知设置', **settings})
return success_response(data=settings)
# ============================================
@@ -223,12 +197,13 @@ class NotificationSettingsView(APIView):
# ============================================
@api_view(['POST'])
@permission_classes([AllowAny]) # Worker 容器无认证,可考虑添加 Token 验
# 权限由全局 IsAuthenticatedOrPublic 处理,/api/callbacks/* 需要 Worker API Key 认
def notification_callback(request):
"""
接收 Worker 的通知推送请求
Worker 容器无法直接访问 Redis通过此 API 回调让 Server 推送 WebSocket。
需要 Worker API Key 认证X-Worker-API-Key Header
POST /api/callbacks/notification/
{
@@ -247,22 +222,24 @@ def notification_callback(request):
required_fields = ['id', 'category', 'title', 'message', 'level', 'created_at']
for field in required_fields:
if field not in data:
return Response(
{'error': f'缺少字段: {field}'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message=f'Missing field: {field}',
status_code=status.HTTP_400_BAD_REQUEST
)
# 推送到 WebSocket
_push_notification_to_websocket(data)
logger.debug(f"回调通知推送成功 - ID: {data['id']}, Title: {data['title']}")
return Response({'status': 'ok'})
return success_response(data={'status': 'ok'})
except Exception as e:
logger.error(f"回调通知处理失败: {e}", exc_info=True)
return Response(
{'error': str(e)},
status=status.HTTP_500_INTERNAL_SERVER_ERROR
return error_response(
code=ErrorCodes.SERVER_ERROR,
message=str(e),
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)

View File

@@ -27,9 +27,6 @@ class BlacklistService:
DEFAULT_PATTERNS = [
r'\.gov$', # .gov 结尾
r'\.gov\.[a-z]{2}$', # .gov.cn, .gov.uk 等
r'\.edu$', # .edu 结尾
r'\.edu\.[a-z]{2}$', # .edu.cn 等
r'\.mil$', # .mil 结尾
]
def __init__(self, patterns: Optional[List[str]] = None):

View File

@@ -113,9 +113,10 @@ def bulk_merge_tech_field(
host = parsed.hostname or ''
# 插入新记录(带冲突处理)
# 显式传入所有 NOT NULL 字段的默认值
insert_sql = f"""
INSERT INTO {table_name} (target_id, url, host, tech, created_at)
VALUES (%s, %s, %s, %s::varchar[], NOW())
INSERT INTO {table_name} (target_id, url, host, location, title, webserver, body_preview, content_type, tech, created_at)
VALUES (%s, %s, %s, '', '', '', '', '', %s::varchar[], NOW())
ON CONFLICT (target_id, url) DO UPDATE SET
tech = (
SELECT ARRAY(SELECT DISTINCT unnest(

View File

@@ -36,7 +36,14 @@ def _normalize_config_keys(config: Dict[str, Any]) -> Dict[str, Any]:
Returns:
key 已转换的新字典
Raises:
ValueError: 配置为 None 或非字典类型时抛出
"""
if config is None:
raise ValueError("配置不能为空None请检查 YAML 格式,确保冒号后有配置内容或使用 {} 表示空配置")
if not isinstance(config, dict):
raise ValueError(f"配置格式错误:期望 dict实际 {type(config).__name__}")
return {
k.replace('-', '_') if isinstance(k, str) else k: v
for k, v in config.items()

View File

@@ -18,6 +18,9 @@ FINGERPRINT_LIB_MAP = {
'ehole': 'ensure_ehole_fingerprint_local',
'goby': 'ensure_goby_fingerprint_local',
'wappalyzer': 'ensure_wappalyzer_fingerprint_local',
'fingers': 'ensure_fingers_fingerprint_local',
'fingerprinthub': 'ensure_fingerprinthub_fingerprint_local',
'arl': 'ensure_arl_fingerprint_local',
}
@@ -221,10 +224,170 @@ def get_fingerprint_paths(lib_names: list) -> dict:
return paths
def ensure_fingers_fingerprint_local() -> str:
"""
确保本地存在最新的 Fingers 指纹文件(带缓存)
Returns:
str: 本地指纹文件路径
"""
from apps.engine.services.fingerprints import FingersFingerprintService
service = FingersFingerprintService()
current_version = service.get_fingerprint_version()
# 缓存目录和文件
base_dir = getattr(settings, 'FINGERPRINTS_BASE_PATH', '/opt/xingrin/fingerprints')
os.makedirs(base_dir, exist_ok=True)
cache_file = os.path.join(base_dir, 'fingers.json')
version_file = os.path.join(base_dir, 'fingers.version')
# 检查缓存版本
cached_version = None
if os.path.exists(version_file):
try:
with open(version_file, 'r') as f:
cached_version = f.read().strip()
except OSError as e:
logger.warning("读取 Fingers 版本文件失败: %s", e)
# 版本匹配,直接返回缓存
if cached_version == current_version and os.path.exists(cache_file):
logger.info("Fingers 指纹文件缓存有效(版本匹配): %s", cache_file)
return cache_file
# 版本不匹配,重新导出
logger.info(
"Fingers 指纹文件需要更新: cached=%s, current=%s",
cached_version, current_version
)
data = service.get_export_data()
with open(cache_file, 'w', encoding='utf-8') as f:
json.dump(data, f, ensure_ascii=False)
# 写入版本文件
try:
with open(version_file, 'w') as f:
f.write(current_version)
except OSError as e:
logger.warning("写入 Fingers 版本文件失败: %s", e)
logger.info("Fingers 指纹文件已更新: %s", cache_file)
return cache_file
def ensure_fingerprinthub_fingerprint_local() -> str:
"""
确保本地存在最新的 FingerPrintHub 指纹文件(带缓存)
Returns:
str: 本地指纹文件路径
"""
from apps.engine.services.fingerprints import FingerPrintHubFingerprintService
service = FingerPrintHubFingerprintService()
current_version = service.get_fingerprint_version()
# 缓存目录和文件
base_dir = getattr(settings, 'FINGERPRINTS_BASE_PATH', '/opt/xingrin/fingerprints')
os.makedirs(base_dir, exist_ok=True)
cache_file = os.path.join(base_dir, 'fingerprinthub.json')
version_file = os.path.join(base_dir, 'fingerprinthub.version')
# 检查缓存版本
cached_version = None
if os.path.exists(version_file):
try:
with open(version_file, 'r') as f:
cached_version = f.read().strip()
except OSError as e:
logger.warning("读取 FingerPrintHub 版本文件失败: %s", e)
# 版本匹配,直接返回缓存
if cached_version == current_version and os.path.exists(cache_file):
logger.info("FingerPrintHub 指纹文件缓存有效(版本匹配): %s", cache_file)
return cache_file
# 版本不匹配,重新导出
logger.info(
"FingerPrintHub 指纹文件需要更新: cached=%s, current=%s",
cached_version, current_version
)
data = service.get_export_data()
with open(cache_file, 'w', encoding='utf-8') as f:
json.dump(data, f, ensure_ascii=False)
# 写入版本文件
try:
with open(version_file, 'w') as f:
f.write(current_version)
except OSError as e:
logger.warning("写入 FingerPrintHub 版本文件失败: %s", e)
logger.info("FingerPrintHub 指纹文件已更新: %s", cache_file)
return cache_file
def ensure_arl_fingerprint_local() -> str:
"""
确保本地存在最新的 ARL 指纹文件(带缓存)
Returns:
str: 本地指纹文件路径YAML 格式)
"""
import yaml
from apps.engine.services.fingerprints import ARLFingerprintService
service = ARLFingerprintService()
current_version = service.get_fingerprint_version()
# 缓存目录和文件
base_dir = getattr(settings, 'FINGERPRINTS_BASE_PATH', '/opt/xingrin/fingerprints')
os.makedirs(base_dir, exist_ok=True)
cache_file = os.path.join(base_dir, 'arl.yaml')
version_file = os.path.join(base_dir, 'arl.version')
# 检查缓存版本
cached_version = None
if os.path.exists(version_file):
try:
with open(version_file, 'r') as f:
cached_version = f.read().strip()
except OSError as e:
logger.warning("读取 ARL 版本文件失败: %s", e)
# 版本匹配,直接返回缓存
if cached_version == current_version and os.path.exists(cache_file):
logger.info("ARL 指纹文件缓存有效(版本匹配): %s", cache_file)
return cache_file
# 版本不匹配,重新导出
logger.info(
"ARL 指纹文件需要更新: cached=%s, current=%s",
cached_version, current_version
)
data = service.get_export_data()
with open(cache_file, 'w', encoding='utf-8') as f:
yaml.dump(data, f, allow_unicode=True, default_flow_style=False)
# 写入版本文件
try:
with open(version_file, 'w') as f:
f.write(current_version)
except OSError as e:
logger.warning("写入 ARL 版本文件失败: %s", e)
logger.info("ARL 指纹文件已更新: %s", cache_file)
return cache_file
__all__ = [
"ensure_ehole_fingerprint_local",
"ensure_goby_fingerprint_local",
"ensure_wappalyzer_fingerprint_local",
"ensure_fingers_fingerprint_local",
"ensure_fingerprinthub_fingerprint_local",
"ensure_arl_fingerprint_local",
"get_fingerprint_paths",
"FINGERPRINT_LIB_MAP",
]

View File

@@ -19,7 +19,6 @@ from typing import Optional
from django.conf import settings
from apps.common.utils.git_proxy import get_git_proxy_url
from apps.engine.models import NucleiTemplateRepo
logger = logging.getLogger(__name__)
@@ -50,7 +49,7 @@ def get_local_commit_hash(local_path: Path) -> Optional[str]:
def git_clone(repo_url: str, local_path: Path) -> bool:
"""Git clone 仓库(支持 Git 加速)
"""Git clone 仓库
Args:
repo_url: 仓库 URL
@@ -59,15 +58,9 @@ def git_clone(repo_url: str, local_path: Path) -> bool:
Returns:
是否成功
"""
# Transform URL for Git acceleration if enabled
proxied_url = get_git_proxy_url(repo_url)
if proxied_url != repo_url:
logger.info("Using Git acceleration: %s -> %s", repo_url, proxied_url)
logger.info("正在 clone 模板仓库: %s -> %s", repo_url, local_path)
result = subprocess.run(
["git", "clone", "--depth", "1", proxied_url, str(local_path)],
["git", "clone", "--depth", "1", repo_url, str(local_path)],
check=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,

View File

@@ -96,7 +96,13 @@ def ensure_wordlist_local(wordlist_name: str) -> str:
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE
with urllib_request.urlopen(download_url, context=ssl_context) as resp:
# 创建带 API Key 的请求
req = urllib_request.Request(download_url)
worker_api_key = os.getenv('WORKER_API_KEY', '')
if worker_api_key:
req.add_header('X-Worker-API-Key', worker_api_key)
with urllib_request.urlopen(req, context=ssl_context) as resp:
if resp.status != 200:
raise RuntimeError(f"下载字典失败HTTP {resp.status}")
data = resp.read()

View File

@@ -7,6 +7,9 @@ from django.core.exceptions import ObjectDoesNotExist, ValidationError
from django.db.utils import DatabaseError, IntegrityError, OperationalError
import logging
from apps.common.response_helpers import success_response, error_response
from apps.common.error_codes import ErrorCodes
logger = logging.getLogger(__name__)
from ..models import Scan, ScheduledScan
@@ -75,20 +78,31 @@ class ScanViewSet(viewsets.ModelViewSet):
scan_service = ScanService()
result = scan_service.delete_scans_two_phase([scan.id])
return Response({
'message': f'已删除扫描任务: Scan #{scan.id}',
'scanId': scan.id,
'deletedCount': result['soft_deleted_count'],
'deletedScans': result['scan_names']
}, status=status.HTTP_200_OK)
return success_response(
data={
'scanId': scan.id,
'deletedCount': result['soft_deleted_count'],
'deletedScans': result['scan_names']
}
)
except Scan.DoesNotExist:
raise NotFound('扫描任务不存在')
return error_response(
code=ErrorCodes.NOT_FOUND,
status_code=status.HTTP_404_NOT_FOUND
)
except ValueError as e:
raise NotFound(str(e))
return error_response(
code=ErrorCodes.NOT_FOUND,
message=str(e),
status_code=status.HTTP_404_NOT_FOUND
)
except Exception as e:
logger.exception("删除扫描任务时发生错误")
raise APIException('服务器错误,请稍后重试')
return error_response(
code=ErrorCodes.SERVER_ERROR,
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
@action(detail=False, methods=['post'])
def quick(self, request):
@@ -132,10 +146,12 @@ class ScanViewSet(viewsets.ModelViewSet):
targets = result['targets']
if not targets:
return Response({
'error': '没有有效的目标可供扫描',
'errors': result.get('errors', [])
}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='No valid targets for scanning',
details=result.get('errors', []),
status_code=status.HTTP_400_BAD_REQUEST
)
# 2. 获取扫描引擎
engine_service = EngineService()
@@ -150,24 +166,44 @@ class ScanViewSet(viewsets.ModelViewSet):
engine=engine
)
# 检查是否成功创建扫描任务
if not created_scans:
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='No scan tasks were created. All targets may already have active scans.',
details={
'targetStats': result['target_stats'],
'assetStats': result['asset_stats'],
'errors': result.get('errors', [])
},
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY
)
# 序列化返回结果
scan_serializer = ScanSerializer(created_scans, many=True)
return Response({
'message': f'快速扫描已启动:{len(created_scans)} 个任务',
'target_stats': result['target_stats'],
'asset_stats': result['asset_stats'],
'errors': result.get('errors', []),
'scans': scan_serializer.data
}, status=status.HTTP_201_CREATED)
return success_response(
data={
'count': len(created_scans),
'targetStats': result['target_stats'],
'assetStats': result['asset_stats'],
'errors': result.get('errors', []),
'scans': scan_serializer.data
},
status_code=status.HTTP_201_CREATED
)
except ValidationError as e:
return Response({'error': str(e)}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message=str(e),
status_code=status.HTTP_400_BAD_REQUEST
)
except Exception as e:
logger.exception("快速扫描启动失败")
return Response(
{'error': '服务器内部错误,请稍后重试'},
status=status.HTTP_500_INTERNAL_SERVER_ERROR
return error_response(
code=ErrorCodes.SERVER_ERROR,
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
@action(detail=False, methods=['post'])
@@ -205,38 +241,47 @@ class ScanViewSet(viewsets.ModelViewSet):
engine=engine
)
# 检查是否成功创建扫描任务
if not created_scans:
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='No scan tasks were created. All targets may already have active scans.',
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY
)
# 序列化返回结果
scan_serializer = ScanSerializer(created_scans, many=True)
return Response(
{
'message': f'已成功发起 {len(created_scans)} 个扫描任务',
return success_response(
data={
'count': len(created_scans),
'scans': scan_serializer.data
},
status=status.HTTP_201_CREATED
status_code=status.HTTP_201_CREATED
)
except ObjectDoesNotExist as e:
# 资源不存在错误(由 service 层抛出)
error_msg = str(e)
return Response(
{'error': error_msg},
status=status.HTTP_404_NOT_FOUND
return error_response(
code=ErrorCodes.NOT_FOUND,
message=str(e),
status_code=status.HTTP_404_NOT_FOUND
)
except ValidationError as e:
# 参数验证错误(由 service 层抛出)
return Response(
{'error': str(e)},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message=str(e),
status_code=status.HTTP_400_BAD_REQUEST
)
except (DatabaseError, IntegrityError, OperationalError):
# 数据库错误
return Response(
{'error': '数据库错误,请稍后重试'},
status=status.HTTP_503_SERVICE_UNAVAILABLE
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Database error',
status_code=status.HTTP_503_SERVICE_UNAVAILABLE
)
# 所有快照相关的 action 和 export 已迁移到 asset/views.py 中的快照 ViewSet
@@ -278,21 +323,24 @@ class ScanViewSet(viewsets.ModelViewSet):
# 参数验证
if not ids:
return Response(
{'error': '缺少必填参数: ids'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='Missing required parameter: ids',
status_code=status.HTTP_400_BAD_REQUEST
)
if not isinstance(ids, list):
return Response(
{'error': 'ids 必须是数组'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='ids must be an array',
status_code=status.HTTP_400_BAD_REQUEST
)
if not all(isinstance(i, int) for i in ids):
return Response(
{'error': 'ids 数组中的所有元素必须是整数'},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message='All elements in ids array must be integers',
status_code=status.HTTP_400_BAD_REQUEST
)
try:
@@ -300,19 +348,27 @@ class ScanViewSet(viewsets.ModelViewSet):
scan_service = ScanService()
result = scan_service.delete_scans_two_phase(ids)
return Response({
'message': f"已删除 {result['soft_deleted_count']} 个扫描任务",
'deletedCount': result['soft_deleted_count'],
'deletedScans': result['scan_names']
}, status=status.HTTP_200_OK)
return success_response(
data={
'deletedCount': result['soft_deleted_count'],
'deletedScans': result['scan_names']
}
)
except ValueError as e:
# 未找到记录
raise NotFound(str(e))
return error_response(
code=ErrorCodes.NOT_FOUND,
message=str(e),
status_code=status.HTTP_404_NOT_FOUND
)
except Exception as e:
logger.exception("批量删除扫描任务时发生错误")
raise APIException('服务器错误,请稍后重试')
return error_response(
code=ErrorCodes.SERVER_ERROR,
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
@action(detail=False, methods=['get'])
def statistics(self, request):
@@ -337,22 +393,25 @@ class ScanViewSet(viewsets.ModelViewSet):
scan_service = ScanService()
stats = scan_service.get_statistics()
return Response({
'total': stats['total'],
'running': stats['running'],
'completed': stats['completed'],
'failed': stats['failed'],
'totalVulns': stats['total_vulns'],
'totalSubdomains': stats['total_subdomains'],
'totalEndpoints': stats['total_endpoints'],
'totalWebsites': stats['total_websites'],
'totalAssets': stats['total_assets'],
})
return success_response(
data={
'total': stats['total'],
'running': stats['running'],
'completed': stats['completed'],
'failed': stats['failed'],
'totalVulns': stats['total_vulns'],
'totalSubdomains': stats['total_subdomains'],
'totalEndpoints': stats['total_endpoints'],
'totalWebsites': stats['total_websites'],
'totalAssets': stats['total_assets'],
}
)
except (DatabaseError, OperationalError):
return Response(
{'error': '数据库错误,请稍后重试'},
status=status.HTTP_503_SERVICE_UNAVAILABLE
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Database error',
status_code=status.HTTP_503_SERVICE_UNAVAILABLE
)
@action(detail=True, methods=['post'])
@@ -383,35 +442,31 @@ class ScanViewSet(viewsets.ModelViewSet):
# 检查是否是状态不允许的问题
scan = scan_service.get_scan(scan_id=pk, prefetch_relations=False)
if scan and scan.status not in [ScanStatus.RUNNING, ScanStatus.INITIATED]:
return Response(
{
'error': f'无法停止扫描:当前状态为 {ScanStatus(scan.status).label}',
'detail': '只能停止运行中或初始化状态的扫描'
},
status=status.HTTP_400_BAD_REQUEST
return error_response(
code=ErrorCodes.BAD_REQUEST,
message=f'Cannot stop scan: current status is {ScanStatus(scan.status).label}',
status_code=status.HTTP_400_BAD_REQUEST
)
# 其他失败原因
return Response(
{'error': '停止扫描失败'},
status=status.HTTP_500_INTERNAL_SERVER_ERROR
return error_response(
code=ErrorCodes.SERVER_ERROR,
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
return Response(
{
'message': f'扫描已停止,已撤销 {revoked_count} 个任务',
'revokedTaskCount': revoked_count
},
status=status.HTTP_200_OK
return success_response(
data={'revokedTaskCount': revoked_count}
)
except ObjectDoesNotExist:
return Response(
{'error': f'扫描 ID {pk} 不存在'},
status=status.HTTP_404_NOT_FOUND
return error_response(
code=ErrorCodes.NOT_FOUND,
message=f'Scan ID {pk} not found',
status_code=status.HTTP_404_NOT_FOUND
)
except (DatabaseError, IntegrityError, OperationalError):
return Response(
{'error': '数据库错误,请稍后重试'},
status=status.HTTP_503_SERVICE_UNAVAILABLE
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Database error',
status_code=status.HTTP_503_SERVICE_UNAVAILABLE
)

View File

@@ -18,6 +18,8 @@ from ..serializers import (
from ..services.scheduled_scan_service import ScheduledScanService
from ..repositories import ScheduledScanDTO
from apps.common.pagination import BasePagination
from apps.common.response_helpers import success_response, error_response
from apps.common.error_codes import ErrorCodes
logger = logging.getLogger(__name__)
@@ -75,15 +77,16 @@ class ScheduledScanViewSet(viewsets.ModelViewSet):
scheduled_scan = self.service.create(dto)
response_serializer = ScheduledScanSerializer(scheduled_scan)
return Response(
{
'message': f'创建定时扫描任务成功: {scheduled_scan.name}',
'scheduled_scan': response_serializer.data
},
status=status.HTTP_201_CREATED
return success_response(
data=response_serializer.data,
status_code=status.HTTP_201_CREATED
)
except ValidationError as e:
return Response({'error': str(e)}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message=str(e),
status_code=status.HTTP_400_BAD_REQUEST
)
def update(self, request, *args, **kwargs):
"""更新定时扫描任务"""
@@ -105,24 +108,27 @@ class ScheduledScanViewSet(viewsets.ModelViewSet):
scheduled_scan = self.service.update(instance.id, dto)
response_serializer = ScheduledScanSerializer(scheduled_scan)
return Response({
'message': f'更新定时扫描任务成功: {scheduled_scan.name}',
'scheduled_scan': response_serializer.data
})
return success_response(data=response_serializer.data)
except ValidationError as e:
return Response({'error': str(e)}, status=status.HTTP_400_BAD_REQUEST)
return error_response(
code=ErrorCodes.VALIDATION_ERROR,
message=str(e),
status_code=status.HTTP_400_BAD_REQUEST
)
def destroy(self, request, *args, **kwargs):
"""删除定时扫描任务"""
instance = self.get_object()
scan_id = instance.id
name = instance.name
if self.service.delete(instance.id):
return Response({
'message': f'删除定时扫描任务成功: {name}',
'id': instance.id
})
return Response({'error': '删除失败'}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
if self.service.delete(scan_id):
return success_response(data={'id': scan_id, 'name': name})
return error_response(
code=ErrorCodes.SERVER_ERROR,
message='Failed to delete scheduled scan',
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR
)
@action(detail=True, methods=['post'])
def toggle(self, request, pk=None):
@@ -136,14 +142,11 @@ class ScheduledScanViewSet(viewsets.ModelViewSet):
scheduled_scan = self.get_object()
response_serializer = ScheduledScanSerializer(scheduled_scan)
status_text = '启用' if is_enabled else '禁用'
return Response({
'message': f'{status_text}定时扫描任务',
'scheduled_scan': response_serializer.data
})
return success_response(data=response_serializer.data)
return Response(
{'error': f'定时扫描任务 ID {pk} 不存在或操作失败'},
status=status.HTTP_404_NOT_FOUND
return error_response(
code=ErrorCodes.NOT_FOUND,
message=f'Scheduled scan with ID {pk} not found or operation failed',
status_code=status.HTTP_404_NOT_FOUND
)

View File

@@ -10,6 +10,7 @@ from .serializers import OrganizationSerializer, TargetSerializer, TargetDetailS
from .services.target_service import TargetService
from .services.organization_service import OrganizationService
from apps.common.pagination import BasePagination
from apps.common.response_helpers import success_response
logger = logging.getLogger(__name__)
@@ -94,9 +95,8 @@ class OrganizationViewSet(viewsets.ModelViewSet):
# 批量解除关联(直接使用 ID避免查询对象
organization.targets.remove(*existing_target_ids)
return Response({
'unlinked_count': existing_count,
'message': f'成功解除 {existing_count} 个目标的关联'
return success_response(data={
'unlinkedCount': existing_count
})
def destroy(self, request, *args, **kwargs):
@@ -124,13 +124,12 @@ class OrganizationViewSet(viewsets.ModelViewSet):
# 直接调用 Service 层的业务方法(软删除 + 分发硬删除任务)
result = self.org_service.delete_organizations_two_phase([organization.id])
return Response({
'message': f'已删除组织: {organization.name}',
return success_response(data={
'organizationId': organization.id,
'organizationName': organization.name,
'deletedCount': result['soft_deleted_count'],
'deletedOrganizations': result['organization_names']
}, status=200)
})
except Organization.DoesNotExist:
raise NotFound('组织不存在')
@@ -181,11 +180,10 @@ class OrganizationViewSet(viewsets.ModelViewSet):
# 调用 Service 层的业务方法(软删除 + 分发硬删除任务)
result = self.org_service.delete_organizations_two_phase(ids)
return Response({
'message': f"已删除 {result['soft_deleted_count']} 个组织",
return success_response(data={
'deletedCount': result['soft_deleted_count'],
'deletedOrganizations': result['organization_names']
}, status=200)
})
except ValueError as e:
raise NotFound(str(e))
@@ -271,12 +269,11 @@ class TargetViewSet(viewsets.ModelViewSet):
# 直接调用 Service 层的业务方法(软删除 + 分发硬删除任务)
result = self.target_service.delete_targets_two_phase([target.id])
return Response({
'message': f'已删除目标: {target.name}',
return success_response(data={
'targetId': target.id,
'targetName': target.name,
'deletedCount': result['soft_deleted_count']
}, status=200)
})
except Target.DoesNotExist:
raise NotFound('目标不存在')
@@ -330,11 +327,10 @@ class TargetViewSet(viewsets.ModelViewSet):
# 调用 Service 层的业务方法(软删除 + 分发硬删除任务)
result = self.target_service.delete_targets_two_phase(ids)
return Response({
'message': f"已删除 {result['soft_deleted_count']} 个目标",
return success_response(data={
'deletedCount': result['soft_deleted_count'],
'deletedTargets': result['target_names']
}, status=200)
})
except ValueError as e:
raise NotFound(str(e))
@@ -389,7 +385,7 @@ class TargetViewSet(viewsets.ModelViewSet):
raise ValidationError(str(e))
# 3. 返回响应
return Response(result, status=status.HTTP_201_CREATED)
return success_response(data=result, status_code=status.HTTP_201_CREATED)
# subdomains action 已迁移到 SubdomainViewSet 嵌套路由
# GET /api/targets/{id}/subdomains/ -> SubdomainViewSet

View File

@@ -177,6 +177,10 @@ STATIC_URL = 'static/'
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
# ==================== Worker API Key 配置 ====================
# Worker 节点认证密钥(从环境变量读取)
WORKER_API_KEY = os.environ.get('WORKER_API_KEY', '')
# ==================== REST Framework 配置 ====================
REST_FRAMEWORK = {
'DEFAULT_PAGINATION_CLASS': 'apps.common.pagination.BasePagination', # 使用基础分页器
@@ -186,6 +190,14 @@ REST_FRAMEWORK = {
'apps.common.authentication.CsrfExemptSessionAuthentication',
],
# 全局权限配置:默认需要认证,公开端点和 Worker 端点在权限类中单独处理
'DEFAULT_PERMISSION_CLASSES': [
'apps.common.permissions.IsAuthenticatedOrPublic',
],
# 自定义异常处理器:统一 401/403 错误响应格式
'EXCEPTION_HANDLER': 'apps.common.exception_handlers.custom_exception_handler',
# JSON 命名格式转换:后端 snake_case ↔ 前端 camelCase
'DEFAULT_RENDERER_CLASSES': (
'djangorestframework_camel_case.render.CamelCaseJSONRenderer', # 响应数据转换为 camelCase
@@ -278,8 +290,9 @@ ENABLE_COMMAND_LOGGING = get_bool_env('ENABLE_COMMAND_LOGGING', True)
# ==================== 数据目录配置(统一使用 /opt/xingrin ====================
# 所有数据目录统一挂载到 /opt/xingrin便于管理和备份
# 扫描工具基础路径
SCAN_TOOLS_BASE_PATH = os.getenv('SCAN_TOOLS_PATH', '/opt/xingrin/tools')
# 扫描工具基础路径worker 容器内,符合 FHS 标准)
# 使用 /opt/xingrin-tools/bin 隔离项目专用扫描工具,避免与系统工具或 Python 包冲突
SCAN_TOOLS_BASE_PATH = os.getenv('SCAN_TOOLS_PATH', '/opt/xingrin-tools/bin')
# 字典文件基础路径
WORDLISTS_BASE_PATH = os.getenv('WORDLISTS_PATH', '/opt/xingrin/wordlists')
@@ -297,9 +310,6 @@ NUCLEI_PUBLIC_TEMPLATES_DIR = os.getenv('NUCLEI_PUBLIC_TEMPLATES_DIR', '/opt/xin
# Nuclei 官方模板仓库地址
NUCLEI_TEMPLATES_REPO_URL = os.getenv('NUCLEI_TEMPLATES_REPO_URL', 'https://github.com/projectdiscovery/nuclei-templates.git')
# Git 镜像加速配置(用于加速 GitHub 仓库克隆)
GIT_MIRROR = os.getenv('GIT_MIRROR', '').strip()
# 对外访问主机与端口(供 Worker 访问 Django 使用)
PUBLIC_HOST = os.getenv('PUBLIC_HOST', 'localhost').strip()
PUBLIC_PORT = os.getenv('PUBLIC_PORT', '8083').strip() # 对外 HTTPS 端口
@@ -357,25 +367,16 @@ HOST_WORDLISTS_DIR = '/opt/xingrin/wordlists'
# ============================================
# Worker 配置中心(任务容器从 /api/workers/config/ 获取)
# ============================================
# Worker 数据库/Redis 地址由 worker_views.py 的 config API 动态返回
# Worker 数据库地址由 worker_views.py 的 config API 动态返回
# 根据请求来源(本地/远程)返回不同的配置:
# - 本地 WorkerDocker 网络内):使用内部服务名postgres, redis
# - 本地 WorkerDocker 网络内):使用内部服务名 postgres
# - 远程 Worker公网访问使用 PUBLIC_HOST
#
# 以下变量仅作为备用/兼容配置,实际配置由 API 动态生成
# 注意Redis 仅在 Server 容器内使用Worker 不需要直接连接 Redis
_db_host = DATABASES['default']['HOST']
_is_internal_db = _db_host in ('postgres', 'localhost', '127.0.0.1')
WORKER_DB_HOST = os.getenv('WORKER_DB_HOST', _db_host)
# 远程 Worker 访问 Redis 的地址(自动推导)
# - 如果 PUBLIC_HOST 是外部 IP → 使用 PUBLIC_HOST
# - 如果 PUBLIC_HOST 是 Docker 内部名 → 使用 redis本地部署
_is_internal_public = PUBLIC_HOST in ('server', 'localhost', '127.0.0.1')
WORKER_REDIS_URL = os.getenv(
'WORKER_REDIS_URL',
'redis://redis:6379/0' if _is_internal_public else f'redis://{PUBLIC_HOST}:6379/0'
)
# 容器内挂载目标路径(统一使用 /opt/xingrin
CONTAINER_RESULTS_MOUNT = '/opt/xingrin/results'
CONTAINER_LOGS_MOUNT = '/opt/xingrin/logs'

View File

@@ -16,7 +16,6 @@ Including another URLconf
"""
from django.contrib import admin
from django.urls import path, include
from rest_framework import permissions
from drf_yasg.views import get_schema_view
from drf_yasg import openapi
@@ -30,7 +29,6 @@ schema_view = get_schema_view(
description="Web 应用侦察工具 API 文档",
),
public=True,
permission_classes=(permissions.AllowAny,),
)
urlpatterns = [

19519
backend/fingerprints/ARL.yaml Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -28,6 +28,10 @@ uvicorn[standard]==0.30.1
# SSH & 远程部署
paramiko>=3.0.0
# Docker 管理
docker>=6.0.0 # Python Docker SDK
packaging>=21.0 # 版本比较
# 测试框架
pytest==8.0.0
pytest-django==4.7.0

View File

@@ -222,6 +222,9 @@ class TestDataGenerator:
self.create_ehole_fingerprints()
self.create_goby_fingerprints()
self.create_wappalyzer_fingerprints()
self.create_fingers_fingerprints()
self.create_fingerprinthub_fingerprints()
self.create_arl_fingerprints()
self.conn.commit()
print("\n✅ 测试数据生成完成!")
@@ -238,6 +241,7 @@ class TestDataGenerator:
tables = [
# 指纹表
'ehole_fingerprint', 'goby_fingerprint', 'wappalyzer_fingerprint',
'fingers_fingerprint', 'fingerprinthub_fingerprint', 'arl_fingerprint',
# 快照表(先删除,因为有外键依赖 scan)
'vulnerability_snapshot', 'host_port_mapping_snapshot', 'directory_snapshot',
'endpoint_snapshot', 'website_snapshot', 'subdomain_snapshot',
@@ -1990,6 +1994,371 @@ class TestDataGenerator:
print(f" ✓ 创建了 {count} 个 Wappalyzer 指纹\n")
def create_fingers_fingerprints(self):
"""创建 Fingers 指纹数据"""
print("🔍 创建 Fingers 指纹...")
cur = self.conn.cursor()
# 应用名称模板(长名称)
name_templates = [
'Apache-HTTP-Server-Web-Application-Platform-Open-Source-Software',
'Nginx-High-Performance-Web-Server-Reverse-Proxy-Load-Balancer',
'Microsoft-IIS-Internet-Information-Services-Windows-Web-Server',
'Tomcat-Java-Servlet-Container-Apache-Application-Server-Platform',
'WordPress-Content-Management-System-Blogging-Platform-PHP-MySQL',
'Drupal-CMS-Content-Management-Framework-PHP-Community-Platform',
'Joomla-Open-Source-CMS-Web-Content-Management-System-Framework',
'Laravel-PHP-Framework-Web-Application-Development-MVC-Pattern',
'Django-Python-Web-Framework-High-Level-MTV-Architecture-Pattern',
'Ruby-on-Rails-Web-Application-Framework-MVC-Convention-Configuration',
'Express-JS-Node-JS-Web-Application-Framework-Minimal-Flexible',
'Spring-Boot-Java-Framework-Microservices-Enterprise-Application',
'ASP-NET-Core-Cross-Platform-Web-Framework-Microsoft-Open-Source',
'React-JavaScript-Library-Building-User-Interfaces-Facebook-Meta',
'Vue-JS-Progressive-JavaScript-Framework-Web-Application-Development',
'Angular-TypeScript-Platform-Framework-Web-Applications-Google',
'jQuery-JavaScript-Library-DOM-Manipulation-Ajax-Event-Handling',
'Bootstrap-CSS-Framework-Responsive-Mobile-First-Web-Development',
'Tailwind-CSS-Utility-First-Framework-Rapid-UI-Development-Tool',
'Docker-Container-Platform-Application-Deployment-Virtualization',
'Kubernetes-Container-Orchestration-Platform-Cloud-Native-Apps',
'Redis-In-Memory-Data-Structure-Store-Database-Cache-Broker',
'MongoDB-Document-NoSQL-Database-Scalable-High-Performance',
'PostgreSQL-Relational-Database-Management-System-Open-Source',
'MySQL-Database-Management-System-Relational-Database-Oracle',
'Elasticsearch-Search-Analytics-Engine-Distributed-RESTful-API',
'RabbitMQ-Message-Broker-Advanced-Message-Queuing-Protocol',
'Jenkins-Automation-Server-Continuous-Integration-Deployment',
'GitLab-DevOps-Platform-Git-Repository-CI-CD-Pipeline-Management',
'Grafana-Observability-Platform-Metrics-Visualization-Dashboard',
]
# 标签模板
tag_options = [
['web-server', 'http', 'apache', 'linux'],
['web-server', 'reverse-proxy', 'nginx', 'high-performance'],
['web-server', 'windows', 'microsoft', 'iis'],
['cms', 'php', 'wordpress', 'blog', 'mysql'],
['cms', 'php', 'drupal', 'content-management'],
['framework', 'php', 'laravel', 'mvc', 'modern'],
['framework', 'python', 'django', 'full-stack'],
['framework', 'ruby', 'rails', 'mvc', 'convention'],
['framework', 'javascript', 'nodejs', 'express', 'backend'],
['framework', 'java', 'spring', 'enterprise', 'microservices'],
['framework', 'dotnet', 'aspnet', 'microsoft', 'cross-platform'],
['library', 'javascript', 'react', 'frontend', 'ui'],
['framework', 'javascript', 'vue', 'progressive', 'reactive'],
['framework', 'typescript', 'angular', 'google', 'spa'],
['database', 'nosql', 'mongodb', 'document', 'json'],
['database', 'relational', 'postgresql', 'sql', 'open-source'],
['database', 'relational', 'mysql', 'sql', 'oracle'],
['cache', 'database', 'redis', 'in-memory', 'key-value'],
['search', 'analytics', 'elasticsearch', 'distributed', 'restful'],
['container', 'docker', 'virtualization', 'deployment'],
]
# 规则模板
rule_templates = [
# favicon hash 规则
[{'method': 'faviconhash', 'favicon': f'-{random.randint(1000000000, 9999999999)}'}],
# keyword 规则
[{'method': 'keyword', 'keyword': ['X-Powered-By', 'Server', 'X-Generator']}],
# 混合规则
[
{'method': 'keyword', 'keyword': ['content="WordPress', 'wp-content/', 'wp-includes/']},
{'method': 'faviconhash', 'favicon': f'-{random.randint(1000000000, 9999999999)}'}
],
# header 规则
[{'method': 'keyword', 'keyword': ['Server: nginx', 'X-Powered-By: PHP']}],
# body 规则
[{'method': 'keyword', 'keyword': ['<meta name="generator"', 'Powered by', 'Built with']}],
]
# 端口模板
port_options = [
[80, 443],
[80, 443, 8080, 8443],
[80, 443, 8000, 8080, 8443],
[3000, 3001, 5000],
[8080, 8081, 8888, 9000],
[443, 8443, 9443],
[], # 空数组
]
count = 0
batch_data = []
for i in range(200): # 生成 200 条 Fingers 指纹
name = f'{random.choice(name_templates)}-{random.randint(1000, 9999)}'
link = f'https://www.example-{random.randint(1000, 9999)}.com'
rule = random.choice(rule_templates)
tag = random.choice(tag_options)
focus = random.choice([True, False])
default_port = random.choice(port_options)
batch_data.append((
name, link, json.dumps(rule), json.dumps(tag), focus, json.dumps(default_port)
))
count += 1
if batch_data:
execute_values(cur, """
INSERT INTO fingers_fingerprint (name, link, rule, tag, focus, default_port, created_at)
VALUES %s
ON CONFLICT (name) DO NOTHING
""", batch_data, template="(%s, %s, %s, %s, %s, %s, NOW())")
print(f" ✓ 创建了 {count} 个 Fingers 指纹\n")
def create_fingerprinthub_fingerprints(self):
"""创建 FingerPrintHub 指纹数据"""
print("🔍 创建 FingerPrintHub 指纹...")
cur = self.conn.cursor()
# FP ID 前缀
fp_id_prefixes = [
'web', 'cms', 'framework', 'server', 'database', 'cache', 'cdn',
'waf', 'load-balancer', 'proxy', 'api', 'admin', 'monitoring'
]
# 应用名称模板
name_templates = [
'Apache-HTTP-Server-Detection-Web-Platform-Fingerprint',
'Nginx-Web-Server-Identification-Reverse-Proxy-Detection',
'WordPress-CMS-Detection-Content-Management-System-Fingerprint',
'Drupal-CMS-Identification-Web-Content-Platform-Detection',
'Joomla-CMS-Detection-Web-Content-Management-Framework',
'Laravel-Framework-Detection-PHP-Web-Application-Platform',
'Django-Framework-Identification-Python-Web-Framework-Detection',
'Spring-Boot-Framework-Detection-Java-Enterprise-Application',
'React-Library-Detection-JavaScript-UI-Framework-Fingerprint',
'Vue-JS-Framework-Detection-Progressive-JavaScript-Platform',
'Angular-Framework-Identification-TypeScript-Web-Platform',
'Docker-Container-Detection-Virtualization-Platform-Fingerprint',
'Kubernetes-Orchestration-Detection-Container-Management-Platform',
'Redis-Cache-Detection-In-Memory-Database-Fingerprint',
'MongoDB-Database-Detection-NoSQL-Document-Store-Platform',
'PostgreSQL-Database-Detection-Relational-Database-System',
'MySQL-Database-Detection-Relational-Database-Management',
'Elasticsearch-Search-Detection-Analytics-Engine-Platform',
'Jenkins-CI-CD-Detection-Automation-Server-Platform',
'GitLab-DevOps-Detection-Version-Control-Platform-System',
'Grafana-Monitoring-Detection-Observability-Platform-Dashboard',
'Prometheus-Monitoring-Detection-Time-Series-Database-System',
'Kibana-Visualization-Detection-Data-Dashboard-Platform',
'Cloudflare-CDN-Detection-Web-Application-Firewall-Platform',
'Akamai-CDN-Detection-Content-Delivery-Network-Platform',
'AWS-CloudFront-CDN-Detection-Amazon-Web-Services-Platform',
'Microsoft-IIS-Detection-Internet-Information-Services-Server',
'Tomcat-Server-Detection-Java-Servlet-Container-Platform',
'JBoss-Server-Detection-Enterprise-Application-Platform',
'WebLogic-Server-Detection-Oracle-Application-Server-Platform',
]
# 作者模板
authors = [
'security-research-team', 'fingerprint-detection-group', 'web-security-lab',
'cyber-threat-intelligence', 'vulnerability-research-team', 'security-automation-team',
'open-source-security', 'community-contributors', 'detection-engineering-team'
]
# 严重程度
severities = ['info', 'low', 'medium', 'high', 'critical']
# metadata 模板
metadata_templates = [
{
'vendor': 'Apache Software Foundation',
'product': 'Apache HTTP Server',
'verified': True,
'max-request': 1,
'shodan-query': 'http.server:"Apache"'
},
{
'vendor': 'Nginx Inc',
'product': 'Nginx Web Server',
'verified': True,
'max-request': 1,
'shodan-query': 'http.server:"nginx"'
},
{
'vendor': 'WordPress',
'product': 'WordPress CMS',
'verified': True,
'max-request': 2,
'fofa-query': 'body="wp-content"'
},
{
'vendor': 'Various',
'product': 'Web Framework',
'verified': False,
'max-request': 1
},
]
# HTTP 规则模板
http_templates = [
[{
'method': 'GET',
'path': ['{{BaseURL}}'],
'matchers': [{
'type': 'word',
'words': ['Server: nginx', 'X-Powered-By'],
'condition': 'or'
}]
}],
[{
'method': 'GET',
'path': ['{{BaseURL}}/admin'],
'matchers': [{
'type': 'status',
'status': [200, 401, 403]
}]
}],
[{
'method': 'GET',
'path': ['{{BaseURL}}'],
'matchers': [{
'type': 'word',
'words': ['wp-content', 'wordpress'],
'part': 'body',
'condition': 'and'
}]
}],
]
# source_file 模板
source_files = [
'fingerprints/web-servers/apache.yaml',
'fingerprints/web-servers/nginx.yaml',
'fingerprints/cms/wordpress.yaml',
'fingerprints/cms/drupal.yaml',
'fingerprints/frameworks/laravel.yaml',
'fingerprints/frameworks/django.yaml',
'fingerprints/frameworks/spring.yaml',
'fingerprints/databases/mongodb.yaml',
'fingerprints/databases/postgresql.yaml',
'fingerprints/cache/redis.yaml',
]
count = 0
batch_data = []
for i in range(200): # 生成 200 条 FingerPrintHub 指纹
fp_id = f'{random.choice(fp_id_prefixes)}-detection-{random.randint(10000, 99999)}'
name = f'{random.choice(name_templates)}-{random.randint(1000, 9999)}'
author = random.choice(authors)
tags = ','.join(random.sample(['web', 'cms', 'framework', 'server', 'detection', 'fingerprint'], random.randint(2, 4)))
severity = random.choice(severities)
metadata = random.choice(metadata_templates).copy()
http = random.choice(http_templates)
source_file = random.choice(source_files)
batch_data.append((
fp_id, name, author, tags, severity,
json.dumps(metadata), json.dumps(http), source_file
))
count += 1
if batch_data:
execute_values(cur, """
INSERT INTO fingerprinthub_fingerprint (
fp_id, name, author, tags, severity, metadata, http, source_file, created_at
)
VALUES %s
ON CONFLICT (fp_id) DO NOTHING
""", batch_data, template="(%s, %s, %s, %s, %s, %s, %s, %s, NOW())")
print(f" ✓ 创建了 {count} 个 FingerPrintHub 指纹\n")
def create_arl_fingerprints(self):
"""创建 ARL 指纹数据"""
print("🔍 创建 ARL 指纹...")
cur = self.conn.cursor()
# 应用名称模板
name_templates = [
'Apache-HTTP-Server-Web-Platform-Application-Server',
'Nginx-High-Performance-Web-Server-Reverse-Proxy',
'Microsoft-IIS-Internet-Information-Services-Server',
'WordPress-Content-Management-System-Blogging-Platform',
'Drupal-Open-Source-CMS-Content-Management-Framework',
'Joomla-Web-Content-Management-System-Framework',
'Laravel-PHP-Web-Application-Framework-MVC-Pattern',
'Django-Python-Web-Framework-MTV-Architecture',
'Spring-Boot-Java-Enterprise-Application-Framework',
'Express-Node-JS-Web-Application-Framework-Minimal',
'React-JavaScript-Library-User-Interface-Components',
'Vue-JS-Progressive-JavaScript-Framework-Reactive',
'Angular-TypeScript-Web-Application-Framework-Google',
'Docker-Container-Platform-Application-Deployment',
'Kubernetes-Container-Orchestration-Cloud-Native',
'Redis-In-Memory-Database-Cache-Message-Broker',
'MongoDB-Document-NoSQL-Database-Scalable-Platform',
'PostgreSQL-Relational-Database-Management-System',
'MySQL-Database-Management-Relational-Database-Oracle',
'Elasticsearch-Search-Analytics-Engine-Distributed',
'Jenkins-Automation-Server-Continuous-Integration',
'GitLab-DevOps-Platform-Git-Repository-CI-CD-Pipeline',
'Grafana-Observability-Metrics-Visualization-Dashboard',
'Prometheus-Monitoring-Time-Series-Database-Alerting',
'RabbitMQ-Message-Broker-AMQP-Protocol-Queue-System',
'Tomcat-Java-Servlet-Container-Application-Server',
'JBoss-Enterprise-Application-Platform-Java-EE-Server',
'WebLogic-Oracle-Application-Server-Java-Enterprise',
'Cloudflare-CDN-DDoS-Protection-Web-Firewall-Platform',
'Amazon-CloudFront-CDN-Content-Delivery-Network-AWS',
]
# 规则表达式模板
rule_templates = [
# 简单规则
'header="Server" && header="nginx"',
'body="WordPress" && body="wp-content"',
'title="Admin Panel" || title="Dashboard"',
'header="X-Powered-By" && header="PHP"',
'body="Powered by" && body="Laravel"',
# 复杂规则
'(header="Server" && header="Apache") || (body="Apache" && title="Apache")',
'(body="wp-content" && body="wp-includes") || (header="X-Powered-By" && header="WordPress")',
'(title="Jenkins" && body="Jenkins") || (header="X-Jenkins" && status="200")',
'(body="Spring" && body="Whitelabel Error Page") || header="X-Application-Context"',
'(body="React" && body="react-dom") || (body="__REACT" && body="reactRoot")',
# 带状态码规则
'status="200" && body="nginx" && title="Welcome to nginx"',
'status="403" && body="Apache" && header="Server"',
'status="401" && header="WWW-Authenticate" && body="Unauthorized"',
# 多条件规则
'header="Server" && (body="PHP" || body="Laravel" || body="Symfony")',
'body="Django" && (header="X-Frame-Options" || body="csrfmiddlewaretoken")',
'(title="GitLab" && body="gitlab") || (header="X-GitLab-Feature-Category")',
# JSON API 规则
'body="{\\"version\\"" && body="api" && header="Content-Type"',
'status="200" && body="swagger" && body="openapi"',
# 错误页面规则
'status="404" && body="Not Found" && body="nginx"',
'status="500" && body="Internal Server Error" && body="Apache"',
]
count = 0
batch_data = []
for i in range(200): # 生成 200 条 ARL 指纹
name = f'{random.choice(name_templates)}-{random.randint(1000, 9999)}'
rule = random.choice(rule_templates)
batch_data.append((name, rule))
count += 1
if batch_data:
execute_values(cur, """
INSERT INTO arl_fingerprint (name, rule, created_at)
VALUES %s
ON CONFLICT (name) DO NOTHING
""", batch_data, template="(%s, %s, NOW())")
print(f" ✓ 创建了 {count} 个 ARL 指纹\n")
class MillionDataGenerator:
"""

View File

@@ -95,6 +95,7 @@ EOF
RESPONSE=$(curl -k -s -X POST \
-H "Content-Type: application/json" \
-H "X-Worker-API-Key: ${WORKER_API_KEY}" \
-d "$REGISTER_DATA" \
"${API_URL}/api/workers/register/" 2>/dev/null)
@@ -116,7 +117,7 @@ if [ -z "$WORKER_ID" ]; then
# 等待 Server 就绪
log "等待 Server 就绪..."
for i in $(seq 1 30); do
if curl -k -s "${API_URL}/api/" > /dev/null 2>&1; then
if curl -k -s -H "X-Worker-API-Key: ${WORKER_API_KEY}" "${API_URL}/api/workers/config/?is_local=${IS_LOCAL}" > /dev/null 2>&1; then
log "${GREEN}Server 已就绪${NC}"
break
fi
@@ -189,6 +190,7 @@ EOF
RESPONSE_FILE=$(mktemp)
HTTP_CODE=$(curl -k -s -o "$RESPONSE_FILE" -w "%{http_code}" -X POST \
-H "Content-Type: application/json" \
-H "X-Worker-API-Key: ${WORKER_API_KEY}" \
-d "$JSON_DATA" \
"${API_URL}/api/workers/${WORKER_ID}/heartbeat/" 2>/dev/null || echo "000")
RESPONSE_BODY=$(cat "$RESPONSE_FILE" 2>/dev/null)

View File

@@ -30,6 +30,7 @@ IMAGE="${DOCKER_USER}/xingrin-agent:${IMAGE_TAG}"
# 预设变量(远程部署时由 deploy_service.py 替换)
PRESET_SERVER_URL="{{HEARTBEAT_API_URL}}"
PRESET_WORKER_ID="{{WORKER_ID}}"
PRESET_API_KEY="{{WORKER_API_KEY}}"
# 颜色定义
GREEN='\033[0;32m'
@@ -68,6 +69,7 @@ start_agent() {
-e SERVER_URL="${PRESET_SERVER_URL}" \
-e WORKER_ID="${PRESET_WORKER_ID}" \
-e IMAGE_TAG="${IMAGE_TAG}" \
-e WORKER_API_KEY="${PRESET_API_KEY}" \
-v /proc:/host/proc:ro \
${IMAGE}

View File

@@ -9,9 +9,8 @@ DB_USER=postgres
DB_PASSWORD=123.com
# ==================== Redis 配置 ====================
# 在 Docker 网络中Redis 服务名称为 redis
# Redis 仅在 Docker 内部网络使用,不暴露公网端口
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
# ==================== 服务端口配置 ====================
@@ -41,6 +40,9 @@ SCAN_RESULTS_DIR=/opt/xingrin/results
# Django 日志目录
# 注意:如果留空或删除此变量,日志将只输出到 Docker 控制台(标准输出),不写入文件
LOG_DIR=/opt/xingrin/logs
# 扫描工具路径(容器内路径,符合 FHS 标准,已隔离避免命名冲突)
# 默认值已在 settings.py 中设置,无需修改,除非需要回退到旧路径
SCAN_TOOLS_PATH=/opt/xingrin-tools/bin
# ==================== 日志级别配置 ====================
# 应用日志级别DEBUG / INFO / WARNING / ERROR
@@ -48,6 +50,12 @@ LOG_LEVEL=INFO
# 是否记录命令执行日志(大量扫描时会增加磁盘占用)
ENABLE_COMMAND_LOGGING=true
# ==================== Worker API Key 配置 ====================
# Worker 节点认证密钥(用于 Worker 与主服务器之间的 API 认证)
# 生产环境务必更换为随机强密钥(建议 32 位以上随机字符串)
# 生成方法: openssl rand -hex 32
WORKER_API_KEY=change-me-to-a-secure-random-key
# ==================== Docker Hub 配置(生产模式) ====================
# 生产模式下从 Docker Hub 拉取镜像时使用
DOCKER_USER=yyhuni

View File

@@ -24,8 +24,6 @@ services:
redis:
image: redis:7-alpine
restart: always
ports:
- "${REDIS_PORT}:6379"
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
@@ -49,7 +47,8 @@ services:
- /opt/xingrin:/opt/xingrin
- /var/run/docker.sock:/var/run/docker.sock
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8888/api/"]
# 使用专门的健康检查端点(无需认证)
test: ["CMD", "curl", "-f", "http://localhost:8888/api/health/"]
interval: 30s
timeout: 10s
retries: 3
@@ -68,6 +67,7 @@ services:
- WORKER_NAME=本地节点
- IS_LOCAL=true
- IMAGE_TAG=${IMAGE_TAG:-dev}
- WORKER_API_KEY=${WORKER_API_KEY}
depends_on:
server:
condition: service_healthy

View File

@@ -30,8 +30,6 @@ services:
redis:
image: redis:7-alpine
restart: always
ports:
- "${REDIS_PORT}:6379"
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
@@ -52,7 +50,8 @@ services:
# Docker Socket 挂载:允许 Django 服务器执行本地 docker 命令(用于本地 Worker 任务分发)
- /var/run/docker.sock:/var/run/docker.sock
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8888/api/"]
# 使用专门的健康检查端点(无需认证)
test: ["CMD", "curl", "-f", "http://localhost:8888/api/health/"]
interval: 30s
timeout: 10s
retries: 3
@@ -72,6 +71,7 @@ services:
- WORKER_NAME=本地节点
- IS_LOCAL=true
- IMAGE_TAG=${IMAGE_TAG}
- WORKER_API_KEY=${WORKER_API_KEY}
depends_on:
server:
condition: service_healthy

View File

@@ -165,7 +165,7 @@ main() {
init_engine_config
init_wordlists
# init_fingerprints
init_fingerprints
init_nuclei_templates
init_admin_user

View File

@@ -23,7 +23,7 @@ echo " ✓ 默认目录字典已就绪"
echo " [1.4/3] 初始化默认指纹库..."
# python manage.py init_fingerprints
python manage.py init_fingerprints
echo " ✓ 默认指纹库已就绪"
# 2. 启动 Django uvicorn 服务 (ASGI)

View File

@@ -1,5 +1,6 @@
# 第一阶段:使用 Go 官方镜像编译工具
FROM golang:1.24 AS go-builder
# 锁定 digest 避免上游更新导致缓存失效
FROM golang:1.24@sha256:7e050c14ae9ca5ae56408a288336545b18632f51402ab0ec8e7be0e649a1fc42 AS go-builder
ENV GOPROXY=https://goproxy.cn,direct
# Naabu 需要 CGO 和 libpcap
@@ -36,7 +37,8 @@ RUN CGO_ENABLED=0 go install -v github.com/owasp-amass/amass/v5/cmd/amass@main
RUN go install github.com/hahwul/dalfox/v2@latest
# 第二阶段:运行时镜像
FROM ubuntu:24.04
# 锁定 digest 避免上游更新导致缓存失效
FROM ubuntu:24.04@sha256:4fdf0125919d24aec972544669dcd7d6a26a8ad7e6561c73d5549bd6db258ac2
# 避免交互式提示
ENV DEBIAN_FRONTEND=noninteractive
@@ -71,20 +73,26 @@ RUN pipx install uro && \
pipx install waymore && \
pipx install dnsgen
# 3. 安装 Sublist3r统一放在 /opt/xingrin/tools 下
RUN git clone https://github.com/aboul3la/Sublist3r.git /opt/xingrin/tools/Sublist3r && \
pip3 install --no-cache-dir -r /opt/xingrin/tools/Sublist3r/requirements.txt --break-system-packages
# 3. 安装 Sublist3rPython 脚本工具,放在 /usr/local/share 标准目录
RUN git clone https://github.com/aboul3la/Sublist3r.git /usr/local/share/Sublist3r && \
pip3 install --no-cache-dir -r /usr/local/share/Sublist3r/requirements.txt --break-system-packages
# 4. 从 go-builder 阶段复制 Go 环境和编译好的工具
# 扫描工具统一放在 /opt/xingrin/tools/,避免与 Python 包冲突(如 httpx
# 创建项目专用工具目录(符合 FHS 标准,/opt 用于独立软件包
# 避免与系统工具或 Python 包冲突,避免被 /opt/xingrin 挂载覆盖
RUN mkdir -p /opt/xingrin-tools/bin
ENV GOPATH=/root/go
ENV PATH=/usr/local/go/bin:/opt/xingrin/tools:$PATH:$GOPATH/bin
ENV GOPROXY=https://goproxy.cn,direct
RUN mkdir -p /opt/xingrin/tools
COPY --from=go-builder /usr/local/go /usr/local/go
COPY --from=go-builder /go/bin/* /opt/xingrin/tools/
COPY --from=go-builder /usr/local/bin/massdns /opt/xingrin/tools/massdns
# 从 go-builder 复制扫描工具到专用目录(避免与系统工具或 Python 包冲突)
COPY --from=go-builder /go/bin/* /opt/xingrin-tools/bin/
COPY --from=go-builder /usr/local/bin/massdns /opt/xingrin-tools/bin/massdns
# 将专用工具目录添加到 PATH优先级高于 /usr/local/bin避免冲突
ENV PATH=/opt/xingrin-tools/bin:/usr/local/go/bin:/usr/local/bin:$PATH:$GOPATH/bin
# 5. 安装 uv Python 包管理器)并安装 Python 依赖
COPY backend/requirements.txt .

View File

@@ -13,21 +13,16 @@
- **权限**: sudo 管理员权限
- **端口要求**: 需要开放以下端口
- `8083` - HTTPS 访问(主要访问端口)
- `5432` - PostgreSQL 数据库(如使用本地数据库)
- `6379` - Redis 缓存服务
- `5432` - PostgreSQL 数据库(如使用本地数据库且有远程 Worker
- 后端 API 仅容器内监听 8888由 nginx 反代到 8083对公网无需放行 8888
- Redis 仅在 Docker 内部网络使用,无需对外开放
## 一键安装
### 1. 下载项目
```bash
# 方式 1Git 克隆(推荐)
git clone https://github.com/你的用户名/xingrin.git
cd xingrin
# 方式 2下载 ZIP
wget https://github.com/你的用户名/xingrin/archive/main.zip
unzip main.zip && cd xingrin-main
```
### 2. 执行安装
@@ -60,8 +55,7 @@ sudo ./install.sh --no-frontend
#### 必须放行的端口
```
8083 - HTTPS 访问(主要访问端口)
5432 - PostgreSQL如使用本地数据库
6379 - Redis 缓存
5432 - PostgreSQL如使用本地数据库且有远程 Worker
```
#### 推荐方案
@@ -110,9 +104,6 @@ graph TD
# 重启服务
./restart.sh
# 更新系统
./update.sh
# 卸载系统
./uninstall.sh
```
@@ -234,11 +225,6 @@ docker logs --tail 100 xingrin-agent
tail -f /opt/xingrin/logs/*.log
```
### 3. 定期更新
```bash
# 定期执行系统更新
./update.sh
```
## 下一步

Binary file not shown.

Before

Width:  |  Height:  |  Size: 95 KiB

After

Width:  |  Height:  |  Size: 112 KiB

View File

@@ -24,6 +24,7 @@ import { Suspense } from "react"
import Script from "next/script"
import { QueryProvider } from "@/components/providers/query-provider"
import { ThemeProvider } from "@/components/providers/theme-provider"
import { UiI18nProvider } from "@/components/providers/ui-i18n-provider"
// Import common layout components
import { RoutePrefetch } from "@/components/route-prefetch"
@@ -117,12 +118,15 @@ export default async function LocaleLayout({
<NextIntlClientProvider messages={messages}>
{/* QueryProvider provides React Query functionality */}
<QueryProvider>
{/* Route prefetch */}
<RoutePrefetch />
{/* AuthLayout handles authentication and sidebar display */}
<AuthLayout>
{children}
</AuthLayout>
{/* UiI18nProvider provides UI component translations */}
<UiI18nProvider>
{/* Route prefetch */}
<RoutePrefetch />
{/* AuthLayout handles authentication and sidebar display */}
<AuthLayout>
{children}
</AuthLayout>
</UiI18nProvider>
</QueryProvider>
</NextIntlClientProvider>
</ThemeProvider>

View File

@@ -95,7 +95,7 @@ export default function ScanEnginePage() {
// Internationalization
const tCommon = useTranslations("common")
const tConfirm = useTranslations("common.confirm")
const tNav = useTranslations("nav")
const tNav = useTranslations("navigation")
const tEngine = useTranslations("scan.engine")
// API Hooks

View File

@@ -64,6 +64,16 @@ export default function ScanHistoryLayout({
<div className="flex items-center justify-between px-4 lg:px-6">
<Tabs value={getActiveTab()} className="w-full">
<TabsList>
<TabsTrigger value="websites" asChild>
<Link href={tabPaths.websites} className="flex items-center gap-0.5">
Websites
{counts.websites > 0 && (
<Badge variant="secondary" className="ml-1.5 h-5 min-w-5 rounded-full px-1.5 text-xs">
{counts.websites}
</Badge>
)}
</Link>
</TabsTrigger>
<TabsTrigger value="subdomain" asChild>
<Link href={tabPaths.subdomain} className="flex items-center gap-0.5">
Subdomains
@@ -74,12 +84,12 @@ export default function ScanHistoryLayout({
)}
</Link>
</TabsTrigger>
<TabsTrigger value="websites" asChild>
<Link href={tabPaths.websites} className="flex items-center gap-0.5">
Websites
{counts.websites > 0 && (
<TabsTrigger value="ip-addresses" asChild>
<Link href={tabPaths["ip-addresses"]} className="flex items-center gap-0.5">
IP Addresses
{counts["ip-addresses"] > 0 && (
<Badge variant="secondary" className="ml-1.5 h-5 min-w-5 rounded-full px-1.5 text-xs">
{counts.websites}
{counts["ip-addresses"]}
</Badge>
)}
</Link>
@@ -104,16 +114,6 @@ export default function ScanHistoryLayout({
)}
</Link>
</TabsTrigger>
<TabsTrigger value="ip-addresses" asChild>
<Link href={tabPaths["ip-addresses"]} className="flex items-center gap-0.5">
IP Addresses
{counts["ip-addresses"] > 0 && (
<Badge variant="secondary" className="ml-1.5 h-5 min-w-5 rounded-full px-1.5 text-xs">
{counts["ip-addresses"]}
</Badge>
)}
</Link>
</TabsTrigger>
<TabsTrigger value="vulnerabilities" asChild>
<Link href={tabPaths.vulnerabilities} className="flex items-center gap-0.5">
Vulnerabilities

View File

@@ -64,12 +64,12 @@ export default function ScheduledScanPage() {
},
cron: {
everyMinute: tScan("cron.everyMinute"),
everyNMinutes: tScan("cron.everyNMinutes"),
everyHour: tScan("cron.everyHour"),
everyNHours: tScan("cron.everyNHours"),
everyDay: tScan("cron.everyDay"),
everyWeek: tScan("cron.everyWeek"),
everyMonth: tScan("cron.everyMonth"),
everyNMinutes: tScan.raw("cron.everyNMinutes") as string,
everyHour: tScan.raw("cron.everyHour") as string,
everyNHours: tScan.raw("cron.everyNHours") as string,
everyDay: tScan.raw("cron.everyDay") as string,
everyWeek: tScan.raw("cron.everyWeek") as string,
everyMonth: tScan.raw("cron.everyMonth") as string,
weekdays: tScan.raw("cron.weekdays") as string[],
},
}), [tColumns, tCommon, tScan])

View File

@@ -138,6 +138,16 @@ export default function TargetLayout({
<div className="flex items-center justify-between px-4 lg:px-6">
<Tabs value={getActiveTab()} className="w-full">
<TabsList>
<TabsTrigger value="websites" asChild>
<Link href={tabPaths.websites} className="flex items-center gap-0.5">
Websites
{counts.websites > 0 && (
<Badge variant="secondary" className="ml-1.5 h-5 min-w-5 rounded-full px-1.5 text-xs">
{counts.websites}
</Badge>
)}
</Link>
</TabsTrigger>
<TabsTrigger value="subdomain" asChild>
<Link href={tabPaths.subdomain} className="flex items-center gap-0.5">
Subdomains
@@ -148,12 +158,12 @@ export default function TargetLayout({
)}
</Link>
</TabsTrigger>
<TabsTrigger value="websites" asChild>
<Link href={tabPaths.websites} className="flex items-center gap-0.5">
Websites
{counts.websites > 0 && (
<TabsTrigger value="ip-addresses" asChild>
<Link href={tabPaths["ip-addresses"]} className="flex items-center gap-0.5">
IP Addresses
{counts["ip-addresses"] > 0 && (
<Badge variant="secondary" className="ml-1.5 h-5 min-w-5 rounded-full px-1.5 text-xs">
{counts.websites}
{counts["ip-addresses"]}
</Badge>
)}
</Link>
@@ -178,16 +188,6 @@ export default function TargetLayout({
)}
</Link>
</TabsTrigger>
<TabsTrigger value="ip-addresses" asChild>
<Link href={tabPaths["ip-addresses"]} className="flex items-center gap-0.5">
IP Addresses
{counts["ip-addresses"] > 0 && (
<Badge variant="secondary" className="ml-1.5 h-5 min-w-5 rounded-full px-1.5 text-xs">
{counts["ip-addresses"]}
</Badge>
)}
</Link>
</TabsTrigger>
<TabsTrigger value="vulnerabilities" asChild>
<Link href={tabPaths.vulnerabilities} className="flex items-center gap-0.5">
Vulnerabilities

View File

@@ -0,0 +1,12 @@
"use client"
import React from "react"
import { ARLFingerprintView } from "@/components/fingerprints"
export default function ARLFingerprintPage() {
return (
<div className="px-4 lg:px-6">
<ARLFingerprintView />
</div>
)
}

View File

@@ -0,0 +1,12 @@
"use client"
import React from "react"
import { FingerPrintHubFingerprintView } from "@/components/fingerprints"
export default function FingerPrintHubFingerprintPage() {
return (
<div className="px-4 lg:px-6">
<FingerPrintHubFingerprintView />
</div>
)
}

View File

@@ -0,0 +1,12 @@
"use client"
import React from "react"
import { FingersFingerprintView } from "@/components/fingerprints"
export default function FingersFingerprintPage() {
return (
<div className="px-4 lg:px-6">
<FingersFingerprintView />
</div>
)
}

View File

@@ -16,13 +16,6 @@ import {
import { useFingerprintStats } from "@/hooks/use-fingerprints"
import { useTranslations } from "next-intl"
// Fingerprint library description
const FINGERPRINT_HELP = `
• EHole: Red team key asset identification tool, supports keyword, favicon hash and other identification methods
• Goby: Attack surface mapping tool, contains a large number of web applications and device fingerprints
• Wappalyzer: Browser extension that can identify the technology stack used by websites
`.trim()
/**
* Fingerprint management layout
* Provides tab navigation to switch between different fingerprint libraries
@@ -41,6 +34,9 @@ export default function FingerprintsLayout({
if (pathname.includes("/ehole")) return "ehole"
if (pathname.includes("/goby")) return "goby"
if (pathname.includes("/wappalyzer")) return "wappalyzer"
if (pathname.includes("/fingers")) return "fingers"
if (pathname.includes("/fingerprinthub")) return "fingerprinthub"
if (pathname.includes("/arl")) return "arl"
return "ehole"
}
@@ -50,6 +46,9 @@ export default function FingerprintsLayout({
ehole: `${basePath}/ehole/`,
goby: `${basePath}/goby/`,
wappalyzer: `${basePath}/wappalyzer/`,
fingers: `${basePath}/fingers/`,
fingerprinthub: `${basePath}/fingerprinthub/`,
arl: `${basePath}/arl/`,
}
// Fingerprint library counts
@@ -57,6 +56,9 @@ export default function FingerprintsLayout({
ehole: stats?.ehole || 0,
goby: stats?.goby || 0,
wappalyzer: stats?.wappalyzer || 0,
fingers: stats?.fingers || 0,
fingerprinthub: stats?.fingerprinthub || 0,
arl: stats?.arl || 0,
}
if (isLoading) {
@@ -123,6 +125,36 @@ export default function FingerprintsLayout({
)}
</Link>
</TabsTrigger>
<TabsTrigger value="fingers" asChild>
<Link href={tabPaths.fingers} className="flex items-center gap-0.5">
Fingers
{counts.fingers > 0 && (
<Badge variant="secondary" className="ml-1.5 h-5 min-w-5 rounded-full px-1.5 text-xs">
{counts.fingers}
</Badge>
)}
</Link>
</TabsTrigger>
<TabsTrigger value="fingerprinthub" asChild>
<Link href={tabPaths.fingerprinthub} className="flex items-center gap-0.5">
FingerPrintHub
{counts.fingerprinthub > 0 && (
<Badge variant="secondary" className="ml-1.5 h-5 min-w-5 rounded-full px-1.5 text-xs">
{counts.fingerprinthub}
</Badge>
)}
</Link>
</TabsTrigger>
<TabsTrigger value="arl" asChild>
<Link href={tabPaths.arl} className="flex items-center gap-0.5">
ARL
{counts.arl > 0 && (
<Badge variant="secondary" className="ml-1.5 h-5 min-w-5 rounded-full px-1.5 text-xs">
{counts.arl}
</Badge>
)}
</Link>
</TabsTrigger>
</TabsList>
</Tabs>
@@ -132,7 +164,7 @@ export default function FingerprintsLayout({
<HelpCircle className="h-4 w-4 text-muted-foreground cursor-help" />
</TooltipTrigger>
<TooltipContent side="right" className="max-w-sm whitespace-pre-line">
{FINGERPRINT_HELP}
{t("helpText")}
</TooltipContent>
</Tooltip>
</TooltipProvider>

View File

@@ -37,7 +37,7 @@ export default function WordlistsPage() {
// Internationalization
const tCommon = useTranslations("common")
const tConfirm = useTranslations("common.confirm")
const tNav = useTranslations("nav")
const tNav = useTranslations("navigation")
const t = useTranslations("pages.wordlists")
const locale = useLocale()

View File

@@ -12,13 +12,25 @@ import { Suspense } from "react"
import { useAuth } from "@/hooks/use-auth"
import { useRouter } from "next/navigation"
// Public routes that don't require authentication
// Public routes that don't require authentication (without locale prefix)
const PUBLIC_ROUTES = ["/login"]
interface AuthLayoutProps {
children: React.ReactNode
}
/**
* Check if the current path is a public route
* Handles internationalized paths like /en/login, /zh/login
*/
function isPublicPath(pathname: string): boolean {
// Remove locale prefix (e.g., /en/login -> /login, /zh/login -> /login)
const pathWithoutLocale = pathname.replace(/^\/[a-z]{2}(?=\/|$)/, '')
return PUBLIC_ROUTES.some((route) =>
pathWithoutLocale === route || pathWithoutLocale.startsWith(`${route}/`)
)
}
/**
* Authentication layout component
* Decides whether to show sidebar based on login status and route
@@ -30,9 +42,7 @@ export function AuthLayout({ children }: AuthLayoutProps) {
const tCommon = useTranslations("common")
// Check if it's a public route (login page)
const isPublicRoute = PUBLIC_ROUTES.some((route) =>
pathname.startsWith(route)
)
const isPublicRoute = isPublicPath(pathname)
// Redirect to login page if not authenticated (useEffect must be before all conditional returns)
React.useEffect(() => {

View File

@@ -309,7 +309,7 @@ export function DashboardDataTable() {
<AlertDialogCancel>{t('common.actions.cancel')}</AlertDialogCancel>
<AlertDialogAction
onClick={confirmStop}
className="bg-chart-2 text-white hover:bg-chart-2/90"
className="bg-primary text-primary-foreground hover:bg-primary/90"
>
{t('scan.stopScan')}
</AlertDialogAction>

View File

@@ -44,12 +44,12 @@ export function DashboardScheduledScans() {
},
cron: {
everyMinute: tScan("cron.everyMinute"),
everyNMinutes: tScan("cron.everyNMinutes"),
everyHour: tScan("cron.everyHour"),
everyNHours: tScan("cron.everyNHours"),
everyDay: tScan("cron.everyDay"),
everyWeek: tScan("cron.everyWeek"),
everyMonth: tScan("cron.everyMonth"),
everyNMinutes: tScan.raw("cron.everyNMinutes") as string,
everyHour: tScan.raw("cron.everyHour") as string,
everyNHours: tScan.raw("cron.everyNHours") as string,
everyDay: tScan.raw("cron.everyDay") as string,
everyWeek: tScan.raw("cron.everyWeek") as string,
everyMonth: tScan.raw("cron.everyMonth") as string,
weekdays: tScan.raw("cron.weekdays") as string[],
},
}), [tColumns, tCommon, tScan])

View File

@@ -80,14 +80,14 @@ export function DirectoriesDataTable({
if (onDownloadAll) {
downloadOptions.push({
key: "all",
label: tDownload("allDirectories"),
label: tDownload("all"),
onClick: onDownloadAll,
})
}
if (onDownloadSelected) {
downloadOptions.push({
key: "selected",
label: tDownload("selectedDirectories"),
label: tDownload("selected"),
onClick: onDownloadSelected,
disabled: (count) => count === 0,
})

View File

@@ -208,7 +208,7 @@ export function createEndpointColumns({
return (
<ExpandableTagList
items={tech}
maxVisible={3}
maxLines={2}
variant="outline"
/>
)
@@ -255,7 +255,7 @@ export function createEndpointColumns({
return (
<ExpandableTagList
items={patterns}
maxVisible={3}
maxLines={2}
variant="secondary"
/>
)

View File

@@ -14,6 +14,7 @@ const ENDPOINT_FILTER_FIELDS: FilterField[] = [
{ key: "host", label: "Host", description: "Hostname" },
{ key: "title", label: "Title", description: "Page title" },
{ key: "status", label: "Status", description: "HTTP status code" },
{ key: "tech", label: "Tech", description: "Technologies" },
]
// Endpoint page filter examples
@@ -21,6 +22,7 @@ const ENDPOINT_FILTER_EXAMPLES = [
'url="/api/*" && status="200"',
'host="api.example.com" || host="admin.example.com"',
'title="Dashboard" && status!="404"',
'tech="php" || tech="wordpress"',
]
interface EndpointsDataTableProps<TData extends { id: number | string }, TValue> {
@@ -99,14 +101,14 @@ export function EndpointsDataTable<TData extends { id: number | string }, TValue
if (onDownloadAll) {
downloadOptions.push({
key: "all",
label: tDownload("allEndpoints"),
label: tDownload("all"),
onClick: onDownloadAll,
})
}
if (onDownloadSelected) {
downloadOptions.push({
key: "selected",
label: tDownload("selectedEndpoints"),
label: tDownload("selected"),
onClick: onDownloadSelected,
disabled: (count) => count === 0,
})

View File

@@ -0,0 +1,86 @@
"use client"
import React from "react"
import { ColumnDef } from "@tanstack/react-table"
import { Checkbox } from "@/components/ui/checkbox"
import { DataTableColumnHeader } from "@/components/ui/data-table/column-header"
import { ExpandableCell, ExpandableMonoCell } from "@/components/ui/data-table/expandable-cell"
import type { ARLFingerprint } from "@/types/fingerprint.types"
interface ColumnOptions {
formatDate: (date: string) => string
}
/**
* Create ARL fingerprint table column definitions
*/
export function createARLFingerprintColumns({
formatDate,
}: ColumnOptions): ColumnDef<ARLFingerprint>[] {
return [
{
id: "select",
header: ({ table }) => (
<Checkbox
checked={
table.getIsAllPageRowsSelected() ||
(table.getIsSomePageRowsSelected() && "indeterminate")
}
onCheckedChange={(value) => table.toggleAllPageRowsSelected(!!value)}
aria-label="Select all"
/>
),
cell: ({ row }) => (
<Checkbox
checked={row.getIsSelected()}
onCheckedChange={(value) => row.toggleSelected(!!value)}
aria-label="Select row"
/>
),
enableSorting: false,
enableHiding: false,
enableResizing: false,
size: 40,
},
{
accessorKey: "name",
meta: { title: "Name" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="Name" />
),
cell: ({ row }) => (
<ExpandableCell value={row.getValue("name")} maxLines={2} />
),
enableResizing: true,
size: 250,
},
{
accessorKey: "rule",
meta: { title: "Rule" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="Rule" />
),
cell: ({ row }) => <ExpandableMonoCell value={row.getValue("rule")} maxLines={3} />,
enableResizing: true,
size: 500,
},
{
accessorKey: "createdAt",
meta: { title: "Created" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="Created" />
),
cell: ({ row }) => {
const date = row.getValue("createdAt") as string
return (
<div className="text-sm text-muted-foreground">
{formatDate(date)}
</div>
)
},
enableResizing: false,
size: 160,
},
]
}

View File

@@ -0,0 +1,248 @@
"use client"
import * as React from "react"
import type { ColumnDef } from "@tanstack/react-table"
import {
IconChevronDown,
IconTrash,
IconDownload,
IconUpload,
IconPlus,
IconSettings,
} from "@tabler/icons-react"
import { Button } from "@/components/ui/button"
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu"
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from "@/components/ui/alert-dialog"
import { UnifiedDataTable } from "@/components/ui/data-table"
import type { FilterField } from "@/components/common/smart-filter-input"
import type { ARLFingerprint } from "@/types/fingerprint.types"
import type { PaginationInfo } from "@/types/common.types"
import { useTranslations } from "next-intl"
const ARL_FILTER_EXAMPLES = [
'name="Apache"',
'name=="Nginx"',
'name="WordPress"',
]
interface ARLFingerprintDataTableProps {
data: ARLFingerprint[]
columns: ColumnDef<ARLFingerprint>[]
onSelectionChange?: (selectedRows: ARLFingerprint[]) => void
filterValue?: string
onFilterChange?: (value: string) => void
isSearching?: boolean
onAddSingle?: () => void
onAddImport?: () => void
onExport?: () => void
onBulkDelete?: () => void
onDeleteAll?: () => void
totalCount?: number
pagination?: { pageIndex: number; pageSize: number }
paginationInfo?: PaginationInfo
onPaginationChange?: (pagination: { pageIndex: number; pageSize: number }) => void
}
export function ARLFingerprintDataTable({
data = [],
columns,
onSelectionChange,
filterValue,
onFilterChange,
isSearching = false,
onAddSingle,
onAddImport,
onExport,
onBulkDelete,
onDeleteAll,
totalCount = 0,
pagination: externalPagination,
paginationInfo,
onPaginationChange,
}: ARLFingerprintDataTableProps) {
const [selectedCount, setSelectedCount] = React.useState(0)
const [exportDialogOpen, setExportDialogOpen] = React.useState(false)
const [bulkDeleteDialogOpen, setBulkDeleteDialogOpen] = React.useState(false)
const [deleteAllDialogOpen, setDeleteAllDialogOpen] = React.useState(false)
const t = useTranslations("tools.fingerprints")
const tCommon = useTranslations("common.actions")
const arlFilterFields: FilterField[] = React.useMemo(() => [
{ key: "name", label: "Name", description: t("filter.arl.name") },
], [t])
const handleSmartSearch = (rawQuery: string) => {
if (onFilterChange) {
onFilterChange(rawQuery)
}
}
const handleSelectionChange = (rows: ARLFingerprint[]) => {
setSelectedCount(rows.length)
onSelectionChange?.(rows)
}
const toolbarRightContent = (
<>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button variant="outline" size="sm">
<IconSettings className="h-4 w-4" />
{t("actions.operations")}
<IconChevronDown className="h-4 w-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-48">
{onExport && (
<DropdownMenuItem onClick={() => setExportDialogOpen(true)}>
<IconDownload className="h-4 w-4" />
{t("actions.exportAll")} (YAML)
</DropdownMenuItem>
)}
<DropdownMenuSeparator />
{onBulkDelete && (
<DropdownMenuItem
onClick={() => setBulkDeleteDialogOpen(true)}
disabled={selectedCount === 0}
className="text-destructive focus:text-destructive"
>
<IconTrash className="h-4 w-4" />
{t("actions.deleteSelected")} ({selectedCount})
</DropdownMenuItem>
)}
{onDeleteAll && (
<DropdownMenuItem
onClick={() => setDeleteAllDialogOpen(true)}
className="text-destructive focus:text-destructive"
>
<IconTrash className="h-4 w-4" />
{t("actions.deleteAll")}
</DropdownMenuItem>
)}
</DropdownMenuContent>
</DropdownMenu>
{(onAddSingle || onAddImport) && (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button size="sm">
<IconPlus className="h-4 w-4" />
{t("actions.addFingerprint")}
<IconChevronDown className="h-4 w-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-40">
{onAddSingle && (
<DropdownMenuItem onClick={onAddSingle}>
<IconPlus className="h-4 w-4" />
{t("actions.addSingle")}
</DropdownMenuItem>
)}
{onAddImport && (
<DropdownMenuItem onClick={onAddImport}>
<IconUpload className="h-4 w-4" />
{t("actions.importFile")} (YAML)
</DropdownMenuItem>
)}
</DropdownMenuContent>
</DropdownMenu>
)}
</>
)
return (
<>
<UnifiedDataTable
data={data}
columns={columns}
getRowId={(row) => String(row.id)}
pagination={externalPagination}
paginationInfo={paginationInfo}
onPaginationChange={onPaginationChange}
searchMode="smart"
searchValue={filterValue}
onSearch={handleSmartSearch}
isSearching={isSearching}
filterFields={arlFilterFields}
filterExamples={ARL_FILTER_EXAMPLES}
onSelectionChange={handleSelectionChange}
showBulkDelete={false}
showAddButton={false}
emptyMessage="No results"
toolbarRight={toolbarRightContent}
/>
<AlertDialog open={exportDialogOpen} onOpenChange={setExportDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>{t("dialogs.exportTitle")}</AlertDialogTitle>
<AlertDialogDescription>
{t("dialogs.exportDesc", { count: totalCount })}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>{tCommon("cancel")}</AlertDialogCancel>
<AlertDialogAction onClick={() => { onExport?.(); setExportDialogOpen(false); }}>
{t("dialogs.confirmExport")}
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
<AlertDialog open={bulkDeleteDialogOpen} onOpenChange={setBulkDeleteDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>{t("dialogs.deleteSelectedTitle")}</AlertDialogTitle>
<AlertDialogDescription>
{t("dialogs.deleteSelectedDesc", { count: selectedCount })}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>{tCommon("cancel")}</AlertDialogCancel>
<AlertDialogAction
onClick={() => { onBulkDelete?.(); setBulkDeleteDialogOpen(false); }}
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
>
{t("dialogs.confirmDelete")}
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
<AlertDialog open={deleteAllDialogOpen} onOpenChange={setDeleteAllDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>{t("dialogs.deleteAllTitle")}</AlertDialogTitle>
<AlertDialogDescription>
{t("dialogs.deleteAllDesc", { count: totalCount })}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>{tCommon("cancel")}</AlertDialogCancel>
<AlertDialogAction
onClick={() => { onDeleteAll?.(); setDeleteAllDialogOpen(false); }}
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
>
{t("dialogs.confirmDelete")}
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</>
)
}

View File

@@ -0,0 +1,150 @@
"use client"
import React, { useEffect } from "react"
import { useForm } from "react-hook-form"
import { toast } from "sonner"
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Label } from "@/components/ui/label"
import { Textarea } from "@/components/ui/textarea"
import {
useCreateARLFingerprint,
useUpdateARLFingerprint,
} from "@/hooks/use-fingerprints"
import type { ARLFingerprint } from "@/types/fingerprint.types"
import { useTranslations } from "next-intl"
interface ARLFingerprintDialogProps {
open: boolean
onOpenChange: (open: boolean) => void
fingerprint?: ARLFingerprint | null
onSuccess?: () => void
}
interface FormData {
name: string
rule: string
}
export function ARLFingerprintDialog({
open,
onOpenChange,
fingerprint,
onSuccess,
}: ARLFingerprintDialogProps) {
const isEdit = !!fingerprint
const t = useTranslations("tools.fingerprints")
const tCommon = useTranslations("common.actions")
const createMutation = useCreateARLFingerprint()
const updateMutation = useUpdateARLFingerprint()
const {
register,
handleSubmit,
reset,
formState: { errors, isSubmitting },
} = useForm<FormData>({
defaultValues: {
name: "",
rule: "",
},
})
useEffect(() => {
if (fingerprint) {
reset({
name: fingerprint.name,
rule: fingerprint.rule,
})
} else {
reset({
name: "",
rule: "",
})
}
}, [fingerprint, reset])
const onSubmit = async (data: FormData) => {
const payload = {
name: data.name.trim(),
rule: data.rule.trim(),
}
try {
if (isEdit && fingerprint) {
await updateMutation.mutateAsync({ id: fingerprint.id, data: payload })
toast.success(t("toast.updateSuccess"))
} else {
await createMutation.mutateAsync(payload)
toast.success(t("toast.createSuccess"))
}
onOpenChange(false)
onSuccess?.()
} catch (error: any) {
toast.error(error.message || (isEdit ? t("toast.updateFailed") : t("toast.createFailed")))
}
}
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent className="sm:max-w-[600px] max-h-[80vh] overflow-y-auto">
<DialogHeader>
<DialogTitle>{isEdit ? t("arl.editTitle") : t("arl.addTitle")}</DialogTitle>
<DialogDescription>
{isEdit ? t("arl.editDesc") : t("arl.addDesc")}
</DialogDescription>
</DialogHeader>
<form onSubmit={handleSubmit(onSubmit)} className="space-y-4">
{/* Name */}
<div className="space-y-2">
<Label htmlFor="name">{t("form.name")} *</Label>
<Input
id="name"
placeholder={t("form.arlNamePlaceholder")}
{...register("name", { required: t("form.nameRequired") })}
/>
{errors.name && (
<p className="text-sm text-destructive">{errors.name.message}</p>
)}
</div>
{/* Rule */}
<div className="space-y-2">
<Label htmlFor="rule">{t("form.arlRule")} *</Label>
<Textarea
id="rule"
placeholder={t("form.arlRulePlaceholder")}
className="font-mono text-sm min-h-[120px]"
{...register("rule", { required: t("form.arlRuleRequired") })}
/>
<p className="text-xs text-muted-foreground">
{t("form.arlRuleHint")}
</p>
{errors.rule && (
<p className="text-sm text-destructive">{errors.rule.message}</p>
)}
</div>
<DialogFooter>
<Button type="button" variant="outline" onClick={() => onOpenChange(false)}>
{tCommon("cancel")}
</Button>
<Button type="submit" disabled={isSubmitting}>
{isSubmitting ? "..." : isEdit ? tCommon("save") : tCommon("create")}
</Button>
</DialogFooter>
</form>
</DialogContent>
</Dialog>
)
}

View File

@@ -0,0 +1,183 @@
"use client"
import React, { useState, useMemo } from "react"
import { AlertTriangle } from "lucide-react"
import { toast } from "sonner"
import { useTranslations, useLocale } from "next-intl"
import {
useARLFingerprints,
useBulkDeleteARLFingerprints,
useDeleteAllARLFingerprints,
} from "@/hooks/use-fingerprints"
import { FingerprintService } from "@/services/fingerprint.service"
import { ARLFingerprintDataTable } from "./arl-fingerprint-data-table"
import { createARLFingerprintColumns } from "./arl-fingerprint-columns"
import { ARLFingerprintDialog } from "./arl-fingerprint-dialog"
import { ImportFingerprintDialog } from "./import-fingerprint-dialog"
import { DataTableSkeleton } from "@/components/ui/data-table-skeleton"
import { getDateLocale } from "@/lib/date-utils"
import type { ARLFingerprint } from "@/types/fingerprint.types"
export function ARLFingerprintView() {
const tFingerprints = useTranslations("tools.fingerprints")
const locale = useLocale()
const [selectedFingerprints, setSelectedFingerprints] = useState<ARLFingerprint[]>([])
const [pagination, setPagination] = useState({ pageIndex: 0, pageSize: 10 })
const [filterQuery, setFilterQuery] = useState("")
const [isSearching, setIsSearching] = useState(false)
const [addDialogOpen, setAddDialogOpen] = useState(false)
const [importDialogOpen, setImportDialogOpen] = useState(false)
const { data, isLoading, isFetching, error, refetch } = useARLFingerprints({
page: pagination.pageIndex + 1,
pageSize: pagination.pageSize,
filter: filterQuery || undefined,
})
const bulkDeleteMutation = useBulkDeleteARLFingerprints()
const deleteAllMutation = useDeleteAllARLFingerprints()
React.useEffect(() => {
if (!isFetching && isSearching) {
setIsSearching(false)
}
}, [isFetching, isSearching])
const handleFilterChange = (value: string) => {
setIsSearching(true)
setFilterQuery(value)
setPagination((prev) => ({ ...prev, pageIndex: 0 }))
}
const formatDate = (dateString: string): string => {
return new Date(dateString).toLocaleString(getDateLocale(locale), {
year: "numeric",
month: "numeric",
day: "numeric",
hour: "2-digit",
minute: "2-digit",
hour12: false,
})
}
const handleExport = async () => {
try {
const blob = await FingerprintService.exportARLFingerprints()
const url = URL.createObjectURL(blob)
const a = document.createElement("a")
a.href = url
a.download = `arl-fingerprints-${Date.now()}.yaml`
document.body.appendChild(a)
a.click()
document.body.removeChild(a)
URL.revokeObjectURL(url)
toast.success(tFingerprints("toast.exportSuccess"))
} catch (error: any) {
toast.error(error.message || tFingerprints("toast.exportFailed"))
}
}
const handleBulkDelete = async () => {
if (selectedFingerprints.length === 0) return
try {
const ids = selectedFingerprints.map((f) => f.id)
const result = await bulkDeleteMutation.mutateAsync(ids)
toast.success(tFingerprints("toast.deleteSuccess", { count: result.deleted }))
setSelectedFingerprints([])
} catch (error: any) {
toast.error(error.message || tFingerprints("toast.deleteFailed"))
}
}
const handleDeleteAll = async () => {
try {
const result = await deleteAllMutation.mutateAsync()
toast.success(tFingerprints("toast.deleteSuccess", { count: result.deleted }))
} catch (error: any) {
toast.error(error.message || tFingerprints("toast.deleteFailed"))
}
}
const columns = useMemo(
() => createARLFingerprintColumns({ formatDate }),
[]
)
const fingerprints: ARLFingerprint[] = useMemo(() => {
if (!data?.results) return []
return data.results
}, [data])
const total = data?.total ?? 0
const page = data?.page ?? 1
const serverPageSize = data?.pageSize ?? 10
const totalPages = data?.totalPages ?? 1
const paginationInfo = useMemo(() => ({
total,
page,
pageSize: serverPageSize,
totalPages,
}), [total, page, serverPageSize, totalPages])
if (error) {
return (
<div className="flex flex-col items-center justify-center py-12">
<div className="rounded-full bg-destructive/10 p-3 mb-4">
<AlertTriangle className="h-10 w-10 text-destructive" />
</div>
<h3 className="text-lg font-semibold mb-2">{tFingerprints("loadFailed")}</h3>
<p className="text-muted-foreground text-center mb-4">
{error.message || tFingerprints("loadError")}
</p>
<button
onClick={() => refetch()}
className="px-4 py-2 bg-primary text-primary-foreground rounded-md hover:bg-primary/90"
>
{tFingerprints("reload")}
</button>
</div>
)
}
if (isLoading && !data) {
return <DataTableSkeleton toolbarButtonCount={3} rows={6} columns={4} />
}
return (
<>
<ARLFingerprintDataTable
data={fingerprints}
columns={columns}
onSelectionChange={setSelectedFingerprints}
filterValue={filterQuery}
onFilterChange={handleFilterChange}
isSearching={isSearching}
onAddSingle={() => setAddDialogOpen(true)}
onAddImport={() => setImportDialogOpen(true)}
onExport={handleExport}
onBulkDelete={handleBulkDelete}
onDeleteAll={handleDeleteAll}
totalCount={data?.total || 0}
pagination={pagination}
paginationInfo={paginationInfo}
onPaginationChange={setPagination}
/>
<ARLFingerprintDialog
open={addDialogOpen}
onOpenChange={setAddDialogOpen}
onSuccess={() => refetch()}
/>
<ImportFingerprintDialog
open={importDialogOpen}
onOpenChange={setImportDialogOpen}
fingerprintType="arl"
acceptedFileTypes=".yaml,.yml,.json"
onSuccess={() => refetch()}
/>
</>
)
}

View File

@@ -5,36 +5,20 @@ import { ColumnDef } from "@tanstack/react-table"
import { Checkbox } from "@/components/ui/checkbox"
import { Badge } from "@/components/ui/badge"
import { DataTableColumnHeader } from "@/components/ui/data-table/column-header"
import { ExpandableCell } from "@/components/ui/data-table/expandable-cell"
import { ChevronDown, ChevronUp } from "lucide-react"
import { useTranslations } from "next-intl"
import type { EholeFingerprint } from "@/types/fingerprint.types"
// Translation type definitions
export interface EholeFingerprintTranslations {
columns: {
cms: string
method: string
location: string
keyword: string
type: string
important: string
created: string
}
actions: {
selectAll: string
selectRow: string
expand: string
collapse: string
}
}
interface ColumnOptions {
formatDate: (date: string) => string
t: EholeFingerprintTranslations
}
/**
* Keyword list cell - displays 3 by default, expandable for more
*/
function KeywordListCell({ keywords, t }: { keywords: string[]; t: EholeFingerprintTranslations }) {
function KeywordListCell({ keywords }: { keywords: string[] }) {
const t = useTranslations("tooltips")
const [expanded, setExpanded] = React.useState(false)
if (!keywords || keywords.length === 0) return <span className="text-muted-foreground">-</span>
@@ -54,9 +38,19 @@ function KeywordListCell({ keywords, t }: { keywords: string[]; t: EholeFingerpr
{hasMore && (
<button
onClick={() => setExpanded(!expanded)}
className="text-xs text-primary hover:underline self-start"
className="text-xs text-primary hover:underline self-start flex items-center gap-1"
>
{expanded ? t.actions.collapse : t.actions.expand}
{expanded ? (
<>
<ChevronUp className="h-3 w-3" />
{t("collapse")}
</>
) : (
<>
<ChevronDown className="h-3 w-3" />
{t("expand")}
</>
)}
</button>
)}
</div>
@@ -68,7 +62,6 @@ function KeywordListCell({ keywords, t }: { keywords: string[]; t: EholeFingerpr
*/
export function createEholeFingerprintColumns({
formatDate,
t,
}: ColumnOptions): ColumnDef<EholeFingerprint>[] {
return [
{
@@ -80,14 +73,14 @@ export function createEholeFingerprintColumns({
(table.getIsSomePageRowsSelected() && "indeterminate")
}
onCheckedChange={(value) => table.toggleAllPageRowsSelected(!!value)}
aria-label={t.actions.selectAll}
aria-label="Select all"
/>
),
cell: ({ row }) => (
<Checkbox
checked={row.getIsSelected()}
onCheckedChange={(value) => row.toggleSelected(!!value)}
aria-label={t.actions.selectRow}
aria-label="Select row"
/>
),
enableSorting: false,
@@ -97,21 +90,21 @@ export function createEholeFingerprintColumns({
},
{
accessorKey: "cms",
meta: { title: t.columns.cms },
meta: { title: "CMS" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title={t.columns.cms} />
<DataTableColumnHeader column={column} title="CMS" />
),
cell: ({ row }) => (
<div className="font-medium">{row.getValue("cms")}</div>
<ExpandableCell value={row.getValue("cms")} maxLines={2} />
),
enableResizing: true,
size: 200,
},
{
accessorKey: "method",
meta: { title: t.columns.method },
meta: { title: "Method" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title={t.columns.method} />
<DataTableColumnHeader column={column} title="Method" />
),
cell: ({ row }) => {
const method = row.getValue("method") as string
@@ -126,9 +119,9 @@ export function createEholeFingerprintColumns({
},
{
accessorKey: "location",
meta: { title: t.columns.location },
meta: { title: "Location" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title={t.columns.location} />
<DataTableColumnHeader column={column} title="Location" />
),
cell: ({ row }) => {
const location = row.getValue("location") as string
@@ -143,19 +136,19 @@ export function createEholeFingerprintColumns({
},
{
accessorKey: "keyword",
meta: { title: t.columns.keyword },
meta: { title: "Keyword" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title={t.columns.keyword} />
<DataTableColumnHeader column={column} title="Keyword" />
),
cell: ({ row }) => <KeywordListCell keywords={row.getValue("keyword") || []} t={t} />,
cell: ({ row }) => <KeywordListCell keywords={row.getValue("keyword") || []} />,
enableResizing: true,
size: 300,
},
{
accessorKey: "type",
meta: { title: t.columns.type },
meta: { title: "Type" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title={t.columns.type} />
<DataTableColumnHeader column={column} title="Type" />
),
cell: ({ row }) => {
const type = row.getValue("type") as string
@@ -167,9 +160,9 @@ export function createEholeFingerprintColumns({
},
{
accessorKey: "isImportant",
meta: { title: t.columns.important },
meta: { title: "Important" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title={t.columns.important} />
<DataTableColumnHeader column={column} title="Important" />
),
cell: ({ row }) => {
const isImportant = row.getValue("isImportant")
@@ -180,9 +173,9 @@ export function createEholeFingerprintColumns({
},
{
accessorKey: "createdAt",
meta: { title: t.columns.created },
meta: { title: "Created" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title={t.columns.created} />
<DataTableColumnHeader column={column} title="Created" />
),
cell: ({ row }) => {
const date = row.getValue("createdAt") as string

View File

@@ -11,7 +11,7 @@ import {
} from "@/hooks/use-fingerprints"
import { FingerprintService } from "@/services/fingerprint.service"
import { EholeFingerprintDataTable } from "./ehole-fingerprint-data-table"
import { createEholeFingerprintColumns, EholeFingerprintTranslations } from "./ehole-fingerprint-columns"
import { createEholeFingerprintColumns } from "./ehole-fingerprint-columns"
import { EholeFingerprintDialog } from "./ehole-fingerprint-dialog"
import { ImportFingerprintDialog } from "./import-fingerprint-dialog"
import { DataTableSkeleton } from "@/components/ui/data-table-skeleton"
@@ -19,30 +19,9 @@ import { getDateLocale } from "@/lib/date-utils"
import type { EholeFingerprint } from "@/types/fingerprint.types"
export function EholeFingerprintView() {
const tColumns = useTranslations("columns")
const tCommon = useTranslations("common")
const tTooltips = useTranslations("tooltips")
const tFingerprints = useTranslations("tools.fingerprints")
const locale = useLocale()
// Build translation object
const translations: EholeFingerprintTranslations = {
columns: {
cms: tColumns("fingerprint.cms"),
method: tColumns("fingerprint.method"),
location: tColumns("endpoint.location"),
keyword: tColumns("fingerprint.keyword"),
type: tColumns("common.type"),
important: tColumns("fingerprint.important"),
created: tColumns("fingerprint.created"),
},
actions: {
selectAll: tCommon("actions.selectAll"),
selectRow: tCommon("actions.selectRow"),
expand: tTooltips("expand"),
collapse: tTooltips("collapse"),
},
}
const [selectedFingerprints, setSelectedFingerprints] = useState<EholeFingerprint[]>([])
const [pagination, setPagination] = useState({ pageIndex: 0, pageSize: 10 })
const [filterQuery, setFilterQuery] = useState("")
@@ -130,8 +109,8 @@ export function EholeFingerprintView() {
// Column definitions
const columns = useMemo(
() => createEholeFingerprintColumns({ formatDate, t: translations }),
[translations]
() => createEholeFingerprintColumns({ formatDate }),
[]
)
// Transform data

View File

@@ -0,0 +1,322 @@
"use client"
import React from "react"
import { ColumnDef } from "@tanstack/react-table"
import { Checkbox } from "@/components/ui/checkbox"
import { Badge } from "@/components/ui/badge"
import { DataTableColumnHeader } from "@/components/ui/data-table/column-header"
import { ExpandableCell, ExpandableMonoCell } from "@/components/ui/data-table/expandable-cell"
import { ChevronDown, ChevronUp } from "lucide-react"
import { useTranslations } from "next-intl"
import type { FingerPrintHubFingerprint } from "@/types/fingerprint.types"
interface ColumnOptions {
formatDate: (date: string) => string
}
/**
* Severity badge with color coding (matching Vulnerabilities style)
*/
function SeverityBadge({ severity }: { severity: string }) {
const severityConfig: Record<string, { className: string }> = {
critical: { className: "bg-[#da3633]/10 text-[#da3633] border border-[#da3633]/20 dark:text-[#f85149]" },
high: { className: "bg-[#d29922]/10 text-[#d29922] border border-[#d29922]/20" },
medium: { className: "bg-[#d4a72c]/10 text-[#d4a72c] border border-[#d4a72c]/20" },
low: { className: "bg-[#238636]/10 text-[#238636] border border-[#238636]/20 dark:text-[#3fb950]" },
info: { className: "bg-[#848d97]/10 text-[#848d97] border border-[#848d97]/20" },
}
const config = severityConfig[severity?.toLowerCase()] || severityConfig.info
return (
<Badge className={config.className}>
{severity || "info"}
</Badge>
)
}
/**
* Tags list cell - displays tags with expand/collapse
*/
function TagListCell({ tags }: { tags: string }) {
const t = useTranslations("tooltips")
const [expanded, setExpanded] = React.useState(false)
if (!tags) return <span className="text-muted-foreground">-</span>
const tagArray = tags.split(",").map(t => t.trim())
const displayTags = expanded ? tagArray : tagArray.slice(0, 3)
const hasMore = tagArray.length > 3
return (
<div className="flex flex-col gap-1">
<div className="flex flex-wrap gap-1">
{displayTags.map((tag, idx) => (
<Badge key={idx} variant="secondary" className="text-xs">
{tag}
</Badge>
))}
</div>
{hasMore && (
<button
onClick={() => setExpanded(!expanded)}
className="text-xs text-primary hover:underline self-start flex items-center gap-1"
>
{expanded ? (
<>
<ChevronUp className="h-3 w-3" />
{t("collapse")}
</>
) : (
<>
<ChevronDown className="h-3 w-3" />
{t("expand")}
</>
)}
</button>
)}
</div>
)
}
/**
* Metadata cell - displays vendor, product, verified and queries
*/
function MetadataCell({ metadata }: { metadata: any }) {
const t = useTranslations("tooltips")
const [expanded, setExpanded] = React.useState(false)
if (!metadata || Object.keys(metadata).length === 0) {
return <span className="text-muted-foreground">-</span>
}
const items: { key: string; value: any }[] = []
Object.entries(metadata).forEach(([key, value]) => {
items.push({ key, value })
})
const displayItems = expanded ? items : items.slice(0, 2)
const hasMore = items.length > 2
return (
<div className="flex flex-col gap-1 w-full">
<div className="font-mono text-xs space-y-0.5 w-full">
{displayItems.map((item, idx) => (
<div key={idx} className={expanded ? "break-all w-full" : "truncate w-full"}>
"{item.key}": {JSON.stringify(item.value)}
</div>
))}
</div>
{hasMore && (
<button
onClick={() => setExpanded(!expanded)}
className="text-xs text-primary hover:underline self-start inline-flex items-center gap-0.5"
>
{expanded ? (
<>
<ChevronUp className="h-3 w-3" />
<span>{t("collapse")}</span>
</>
) : (
<>
<ChevronDown className="h-3 w-3" />
<span>{t("expand")}</span>
</>
)}
</button>
)}
</div>
)
}
/**
* HTTP matchers cell - displays detailed HTTP rules in JSON format
*/
function HttpMatchersCell({ http }: { http: any[] }) {
const t = useTranslations("tooltips")
const [expanded, setExpanded] = React.useState(false)
if (!http || http.length === 0) return <span className="text-muted-foreground">-</span>
// Extract key fields from http matchers
const httpItems: { key: string; value: any }[] = []
const hasMultiple = http.length > 1
http.forEach((item, idx) => {
const prefix = hasMultiple ? `[${idx}].` : ""
if (item.path) httpItems.push({ key: `${prefix}path`, value: item.path })
if (item.method) httpItems.push({ key: `${prefix}method`, value: item.method })
if (item.matchers) httpItems.push({ key: `${prefix}matchers`, value: item.matchers })
})
const displayItems = expanded ? httpItems : httpItems.slice(0, 2)
const hasMore = httpItems.length > 2
return (
<div className="flex flex-col gap-1 w-full">
<div className="font-mono text-xs space-y-0.5 w-full">
{displayItems.map((item, idx) => (
<div key={idx} className={expanded ? "break-all w-full" : "truncate w-full"}>
"{item.key}": {JSON.stringify(item.value)}
</div>
))}
</div>
{hasMore && (
<button
onClick={() => setExpanded(!expanded)}
className="text-xs text-primary hover:underline self-start inline-flex items-center gap-0.5"
>
{expanded ? (
<>
<ChevronUp className="h-3 w-3" />
<span>{t("collapse")}</span>
</>
) : (
<>
<ChevronDown className="h-3 w-3" />
<span>{t("expand")}</span>
</>
)}
</button>
)}
</div>
)
}
/**
* Create FingerPrintHub fingerprint table column definitions
*/
export function createFingerPrintHubFingerprintColumns({
formatDate,
}: ColumnOptions): ColumnDef<FingerPrintHubFingerprint>[] {
return [
{
id: "select",
header: ({ table }) => (
<Checkbox
checked={
table.getIsAllPageRowsSelected() ||
(table.getIsSomePageRowsSelected() && "indeterminate")
}
onCheckedChange={(value) => table.toggleAllPageRowsSelected(!!value)}
aria-label="Select all"
/>
),
cell: ({ row }) => (
<Checkbox
checked={row.getIsSelected()}
onCheckedChange={(value) => row.toggleSelected(!!value)}
aria-label="Select row"
/>
),
enableSorting: false,
enableHiding: false,
enableResizing: false,
size: 40,
},
{
accessorKey: "fpId",
meta: { title: "FP ID" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="FP ID" />
),
cell: ({ row }) => (
<ExpandableMonoCell value={row.getValue("fpId")} maxLines={1} />
),
enableResizing: true,
size: 200,
},
{
accessorKey: "name",
meta: { title: "Name" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="Name" />
),
cell: ({ row }) => (
<ExpandableCell value={row.getValue("name")} maxLines={2} />
),
enableResizing: true,
size: 200,
},
{
accessorKey: "author",
meta: { title: "Author" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="Author" />
),
cell: ({ row }) => (
<ExpandableCell value={row.getValue("author")} maxLines={1} />
),
enableResizing: true,
size: 120,
},
{
accessorKey: "severity",
meta: { title: "Severity" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="Severity" />
),
cell: ({ row }) => <SeverityBadge severity={row.getValue("severity")} />,
enableResizing: false,
size: 100,
},
{
accessorKey: "tags",
meta: { title: "Tags" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="Tags" />
),
cell: ({ row }) => <TagListCell tags={row.getValue("tags") || ""} />,
enableResizing: true,
size: 200,
},
{
accessorKey: "metadata",
meta: { title: "Metadata" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="Metadata" />
),
cell: ({ row }) => <MetadataCell metadata={row.getValue("metadata") || {}} />,
enableResizing: true,
size: 300,
},
{
accessorKey: "http",
meta: { title: "HTTP" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="HTTP" />
),
cell: ({ row }) => <HttpMatchersCell http={row.getValue("http") || []} />,
enableResizing: true,
size: 350,
},
{
accessorKey: "sourceFile",
meta: { title: "Source File" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="Source File" />
),
cell: ({ row }) => (
<ExpandableMonoCell value={row.getValue("sourceFile")} maxLines={1} />
),
enableResizing: true,
size: 200,
},
{
accessorKey: "createdAt",
meta: { title: "Created" },
header: ({ column }) => (
<DataTableColumnHeader column={column} title="Created" />
),
cell: ({ row }) => {
const date = row.getValue("createdAt") as string
return (
<div className="text-sm text-muted-foreground">
{formatDate(date)}
</div>
)
},
enableResizing: false,
size: 160,
},
]
}

View File

@@ -0,0 +1,253 @@
"use client"
import * as React from "react"
import type { ColumnDef } from "@tanstack/react-table"
import {
IconChevronDown,
IconTrash,
IconDownload,
IconUpload,
IconPlus,
IconSettings,
} from "@tabler/icons-react"
import { Button } from "@/components/ui/button"
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu"
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from "@/components/ui/alert-dialog"
import { UnifiedDataTable } from "@/components/ui/data-table"
import type { FilterField } from "@/components/common/smart-filter-input"
import type { FingerPrintHubFingerprint } from "@/types/fingerprint.types"
import type { PaginationInfo } from "@/types/common.types"
import { useTranslations } from "next-intl"
const FINGERPRINTHUB_FILTER_EXAMPLES = [
'name="Apache"',
'severity="high"',
'author="pdteam"',
'fpId="apache-detect"',
]
interface FingerPrintHubFingerprintDataTableProps {
data: FingerPrintHubFingerprint[]
columns: ColumnDef<FingerPrintHubFingerprint>[]
onSelectionChange?: (selectedRows: FingerPrintHubFingerprint[]) => void
filterValue?: string
onFilterChange?: (value: string) => void
isSearching?: boolean
onAddSingle?: () => void
onAddImport?: () => void
onExport?: () => void
onBulkDelete?: () => void
onDeleteAll?: () => void
totalCount?: number
pagination?: { pageIndex: number; pageSize: number }
paginationInfo?: PaginationInfo
onPaginationChange?: (pagination: { pageIndex: number; pageSize: number }) => void
}
export function FingerPrintHubFingerprintDataTable({
data = [],
columns,
onSelectionChange,
filterValue,
onFilterChange,
isSearching = false,
onAddSingle,
onAddImport,
onExport,
onBulkDelete,
onDeleteAll,
totalCount = 0,
pagination: externalPagination,
paginationInfo,
onPaginationChange,
}: FingerPrintHubFingerprintDataTableProps) {
const [selectedCount, setSelectedCount] = React.useState(0)
const [exportDialogOpen, setExportDialogOpen] = React.useState(false)
const [bulkDeleteDialogOpen, setBulkDeleteDialogOpen] = React.useState(false)
const [deleteAllDialogOpen, setDeleteAllDialogOpen] = React.useState(false)
const t = useTranslations("tools.fingerprints")
const tCommon = useTranslations("common.actions")
// FingerPrintHub filter field configuration
const filterFields: FilterField[] = React.useMemo(() => [
{ key: "fpId", label: "ID", description: t("filter.fingerprinthub.fpId") },
{ key: "name", label: "Name", description: t("filter.fingerprinthub.name") },
{ key: "author", label: "Author", description: t("filter.fingerprinthub.author") },
{ key: "severity", label: "Severity", description: t("filter.fingerprinthub.severity") },
], [t])
const handleSmartSearch = (rawQuery: string) => {
if (onFilterChange) {
onFilterChange(rawQuery)
}
}
const handleSelectionChange = (rows: FingerPrintHubFingerprint[]) => {
setSelectedCount(rows.length)
onSelectionChange?.(rows)
}
const toolbarRightContent = (
<>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button variant="outline" size="sm">
<IconSettings className="h-4 w-4" />
{t("actions.operations")}
<IconChevronDown className="h-4 w-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-48">
{onExport && (
<DropdownMenuItem onClick={() => setExportDialogOpen(true)}>
<IconDownload className="h-4 w-4" />
{t("actions.exportAll")}
</DropdownMenuItem>
)}
<DropdownMenuSeparator />
{onBulkDelete && (
<DropdownMenuItem
onClick={() => setBulkDeleteDialogOpen(true)}
disabled={selectedCount === 0}
className="text-destructive focus:text-destructive"
>
<IconTrash className="h-4 w-4" />
{t("actions.deleteSelected")} ({selectedCount})
</DropdownMenuItem>
)}
{onDeleteAll && (
<DropdownMenuItem
onClick={() => setDeleteAllDialogOpen(true)}
className="text-destructive focus:text-destructive"
>
<IconTrash className="h-4 w-4" />
{t("actions.deleteAll")}
</DropdownMenuItem>
)}
</DropdownMenuContent>
</DropdownMenu>
{(onAddSingle || onAddImport) && (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button size="sm">
<IconPlus className="h-4 w-4" />
{t("actions.addFingerprint")}
<IconChevronDown className="h-4 w-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-40">
{onAddSingle && (
<DropdownMenuItem onClick={onAddSingle}>
<IconPlus className="h-4 w-4" />
{t("actions.addSingle")}
</DropdownMenuItem>
)}
{onAddImport && (
<DropdownMenuItem onClick={onAddImport}>
<IconUpload className="h-4 w-4" />
{t("actions.importFile")}
</DropdownMenuItem>
)}
</DropdownMenuContent>
</DropdownMenu>
)}
</>
)
return (
<>
<UnifiedDataTable
data={data}
columns={columns}
getRowId={(row) => String(row.id)}
pagination={externalPagination}
paginationInfo={paginationInfo}
onPaginationChange={onPaginationChange}
searchMode="smart"
searchValue={filterValue}
onSearch={handleSmartSearch}
isSearching={isSearching}
filterFields={filterFields}
filterExamples={FINGERPRINTHUB_FILTER_EXAMPLES}
onSelectionChange={handleSelectionChange}
showBulkDelete={false}
showAddButton={false}
emptyMessage="No results"
toolbarRight={toolbarRightContent}
/>
<AlertDialog open={exportDialogOpen} onOpenChange={setExportDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>{t("dialogs.exportTitle")}</AlertDialogTitle>
<AlertDialogDescription>
{t("dialogs.exportDesc", { count: totalCount })}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>{tCommon("cancel")}</AlertDialogCancel>
<AlertDialogAction onClick={() => { onExport?.(); setExportDialogOpen(false); }}>
{t("dialogs.confirmExport")}
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
<AlertDialog open={bulkDeleteDialogOpen} onOpenChange={setBulkDeleteDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>{t("dialogs.deleteSelectedTitle")}</AlertDialogTitle>
<AlertDialogDescription>
{t("dialogs.deleteSelectedDesc", { count: selectedCount })}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>{tCommon("cancel")}</AlertDialogCancel>
<AlertDialogAction
onClick={() => { onBulkDelete?.(); setBulkDeleteDialogOpen(false); }}
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
>
{t("dialogs.confirmDelete")}
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
<AlertDialog open={deleteAllDialogOpen} onOpenChange={setDeleteAllDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>{t("dialogs.deleteAllTitle")}</AlertDialogTitle>
<AlertDialogDescription>
{t("dialogs.deleteAllDesc", { count: totalCount })}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>{tCommon("cancel")}</AlertDialogCancel>
<AlertDialogAction
onClick={() => { onDeleteAll?.(); setDeleteAllDialogOpen(false); }}
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
>
{t("dialogs.confirmDelete")}
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</>
)
}

Some files were not shown because too many files have changed in this diff Show More